You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2008/11/13 21:36:44 UTC
[jira] Created: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
FileSystem.CACHE should be ref-counted
--------------------------------------
Key: HADOOP-4655
URL: https://issues.apache.org/jira/browse/HADOOP-4655
Project: Hadoop Core
Issue Type: Bug
Reporter: Hong Tang
FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650274#action_12650274 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4655:
------------------------------------------------
> I'm not yet convinced this is a bug that needs fixing.
I also don't think this is a bug. This issue is proposing a functional change, which breaks compatibility for all the client codes since FileSystem.get(..) and FileSystem.close() are used everywhere.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682472#action_12682472 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
> Except we don't add new features to branches, but only to trunk, right?
Sounds perfect to me. It is just that 0.20 is an "unreleased" branch.
> Also, what's the use case for newInstanceLocal()?
An application that uses newInstance() method assumes that a new filesystem object is created (irrespective of the path that is being passed in). It does not matter whether it is a local path or a hdfs path. The application logic does not have to change depending on the FileSystem that it is trying to access. So, I would like to keep the newInstanceLocal() method.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Attachment: FileSystemUnique.patch
Here is a small patch that introduces FileSystem.getUnique() to return a unique file system object per invocation. I will write a unit test once people are comfortable with this new API.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680261#action_12680261 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
Read through the java part of the patch, looks mostly good to me. Two questions:
- What should be the semantics of FileSystem.equals()? Should two unique instances pointing to the same File System be considered equal or not?
- There seems to be a redundant line of import of atomic.AtomicLong
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4655:
-------------------------------------------
Component/s: fs
Hadoop Flags: [Incompatible change]
If we do this, this is going to be an incompatible change.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681146#action_12681146 ]
Hadoop QA commented on HADOOP-4655:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401983/FileSystemUnique2.txt
against trunk revision 752682.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/52/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/52/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/52/console
This message is automatically generated.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677578#action_12677578 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
I would rather not deprecate the method that uses the CACHE because that behaviour might be optimal for some class of applications, do you agree?
I will take Hong's suggestion and change the name of this new API to FileSystem.newInstance().
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650315#action_12650315 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
If an application wants to release all the resources associated with an "open" filesystem, it might want to use FileSystem.close(), isn't it? This might be worthwhile for an application that opens many many different FileSystems (especially with different usernames) and then closes them appropriately to release the amount of resources tied up with each open instance.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650310#action_12650310 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
>'Yes, closing a FileSystem that's in use by other threads can break things. But what harm is this causing in practice?
My patke is that this patch makes it easier for an application program to use the FileSystem object. RefCounting the objects make it easier for users of this API to use FileSystem object: they need not know about the existence of the cache. Maybe we can label it as a "improvement" rather than a "bug".
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680216#action_12680216 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
> Changed name of new API to FileSystem.createNewInstance().
A nit: that seems redundant. I'd rather see this as just 'create()' or 'newInstance()', but not both.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648102#action_12648102 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
+1 for this change.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Attachment: FileSystemUnique2.txt
Changed name to FileSystem.newInstance(). Remove redundant imports of AtomicLong.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated HADOOP-4655:
------------------------------
Component/s: dfs
Affects Version/s: 0.20.0
0.19.1
0.18.1
0.18.2
0.19.0
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647407#action_12647407 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
Specifically, if the cache is ref-counted, fs1.close() should not close the actual file system, but it needs to decrement the ref-count in the cached entry.
Also, maybe we need to make FileSystem.close() final, and add a protected abstract method called FileSystem.realClose() that is supposed to be implemented by actual file systems.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-4655:
----------------------------------------
Assignee: dhruba borthakur
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650342#action_12650342 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4655:
------------------------------------------------
"ref-counted" here indeed means "get-counted" since it is not counting the number of references but the number of FileSystem.get(...). It may be a nice thing have but the cost of doing this is very high.
@Houng> the bug could also affect single threaded program
I would argue that the bug is in depthFirstSearch(..) but not FileSystem cache.
@dhruba> This might be worthwhile for an application that opens many many different FileSystems (especially with different usernames)
For this kind of special applications, ref-counted (or get-counted) could be implemented there.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Status: Open (was: Patch Available)
Cancelling patch to incorporate review comments.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.19.1, 0.19.0, 0.18.2, 0.18.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677593#action_12677593 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
In most situations cache is useful. But I don't like the fact that currently all concrete FileSystem implementations must call super.close(). From API symmetry point of view, if you get a FileSystem instance through FS.newInstance(), then you are responsible for calling fs.close(); or if you get an instance from cache as in FS.get(), you should call something like FS.return() instead of fs.close(). If this is enforced, then we can ref-count the CACHE, and FileSystem.close() should not deal with CACHE at all.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650314#action_12650314 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
> the current Hadoop code, the practice is not to call fs.close()
That's correct. So the question is whether we have a strong need to change that.
> Then what is the reason of making FileSystem implement Closeable in the first place?
That is the root question here. If there's no need to explicitly close FileSystem implementations, then perhaps the change is simply to remove FileSystem#close.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647389#action_12647389 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
The following code will either result in resource leakage or invalid FileSystem object:
Path path;
...
FileSystem fs1 = path.getFileSystem(conf);
...
FileSystem fs2 = path.getFileSystem(conf);
// fs1 should be the same as fs2.
...
fs1.close();
...
FileSystem fs3 = path.getFileSystem(conf);
// fs3 would be a newly created FileSystem object (instead of reusing fs1).
Even worse, if the actual FileSystem implemnetation's close() actually invalidates the object (such as DistributedFileSystem), then fs2 would no longer usable after fs1.close().
This is probably related to HADOOP-319.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12684006#action_12684006 ]
Hudson commented on HADOOP-4655:
--------------------------------
Integrated in Hadoop-trunk #785 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/785/])
. New method FileSystem.newInstance() that always returns
a newly allocated FileSystem object. (dhruba)
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650256#action_12650256 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
I'm not yet convinced this is a bug that needs fixing. Yes, closing a FileSystem that's in use by other threads can break things. But what harm is this causing in practice?
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650362#action_12650362 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
bq. If an application wants to release all the resources associated with an "open" filesystem, it might want to use FileSystem.close(), isn't it? This might be worthwhile for an application that opens many many different FileSystems (especially with different usernames) and then closes them appropriately to release the amount of resources tied up with each open instance.
+1
Resource leakage should be considered bugs. Even though the leakage is so slow that it causes little impact in practice does not change that fact. And as Dhruba pointed out, it does not preclude that in the future, some applications may cause much faster leakage, and thus make this a severe issue.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Fix Version/s: 0.20.0
Status: Patch Available (was: Open)
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.19.1, 0.19.0, 0.18.2, 0.18.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680244#action_12680244 ]
Hadoop QA commented on HADOOP-4655:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401565/FileSystemUnique.txt
against trunk revision 751463.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/36/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/36/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/36/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/36/console
This message is automatically generated.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682478#action_12682478 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
> 0.20 is an "unreleased" branch.
True, but the branch date is also the feature freeze-date, no? Documentation, tests and bugfixes are added after the branch date, but not features.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649987#action_12649987 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
I have a better idea to fix the current problem with minimum change of the API:
- Make FileSystem.CACHE ref-counted.
- FileSystem.close() is no-op, and knows nothing about FileSystem.CACHE.
- Add two static methods whose names clearly reflect the shared nature of the FileSystem objects:
- static FileSystem getFileSystem(Path, Configuration)
- static void returnFileSystem(FileSystem)
- Deprecate Path.getFileSystem(Configuration), and suggest to use FileSystem.borrowFileSystem instead.
The recommended usage pattern would be:
- If a FileSystem object is created by new, then people should release resources by FileSystem.close().
- If a FileSystem object is obtained by getFileSystem, then the resource should be released through returnFileSystem.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647400#action_12647400 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4655:
------------------------------------------------
If you do ref-count, does fs1.close() above become a no-op?
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.20.0)
0.21.0
Release Note: Every invocation of FileSystem.newInstance() returns a newly allocated FileSystem object.
Status: Resolved (was: Patch Available)
I have committed this.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-4655) FileSystem.CACHE should
be ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649987#action_12649987 ]
hong.tang edited comment on HADOOP-4655 at 11/22/08 4:16 PM:
-------------------------------------------------------------
I have a better idea to fix the current problem with minimum change of the API:
- Make FileSystem.CACHE ref-counted.
- FileSystem.close() is no-op, and knows nothing about FileSystem.CACHE.
- Add two static methods whose names clearly reflect the shared nature of the FileSystem objects:
- static FileSystem getFileSystem(Path, Configuration)
- static void returnFileSystem(FileSystem)
- Deprecate Path.getFileSystem(Configuration), and suggest to use FileSystem.getFileSystem instead.
The recommended usage pattern would be:
- If a FileSystem object is created by new, then people should release resources by FileSystem.close().
- If a FileSystem object is obtained by getFileSystem, then the resource should be released through returnFileSystem.
was (Author: hong.tang):
I have a better idea to fix the current problem with minimum change of the API:
- Make FileSystem.CACHE ref-counted.
- FileSystem.close() is no-op, and knows nothing about FileSystem.CACHE.
- Add two static methods whose names clearly reflect the shared nature of the FileSystem objects:
- static FileSystem getFileSystem(Path, Configuration)
- static void returnFileSystem(FileSystem)
- Deprecate Path.getFileSystem(Configuration), and suggest to use FileSystem.borrowFileSystem instead.
The recommended usage pattern would be:
- If a FileSystem object is created by new, then people should release resources by FileSystem.close().
- If a FileSystem object is obtained by getFileSystem, then the resource should be released through returnFileSystem.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677576#action_12677576 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
Do we want to deprecate the method that uses the cache? If so, we'd need to make FileSystem instantiation more lightweight. Currently, e.g., HDFS makes a call to the namenode to check version compatibility each time a DistributedFileSystem is created. We'd probably want to cache that and/or do it lazily.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677254#action_12677254 ]
dhruba borthakur commented on HADOOP-4655:
------------------------------------------
I would like to resurrect the discussion on this issue.
I am in the process of integrating a distributed log aggregation service called Scribe (http://developers.facebook.com/scribe/) to be able to store directly into HDFS. The scribe server invokes HDFS APIs using libhdfs. Scribe is a multi-threaded software module, each thread should be able to independently open files from different hdfs clusters. It will be very convenient for scribe to be able to invoke hdfsConnect() just before every time a file is opened. Similarly, after closing a file, it would invoke hdfsdisConnect(). This reduces complexity in the scribe-hdfs integration to a large extent. However, with the current caching behaviour of FileSystem.get(), the above is not possible. Scribe has to maintain reference counts for each open and understand the caching behaviour of FileSystem objects.
I would like to implement the second option that Doug has suggested, i.e. "remove the cache, and allocate a new FileSystem instance per call to getFileSystem()". However, this will be an incompatible change, especially because pre-existing applications might be rely on this singular behavour to elegantly open file systems. To ensure than existing implementations do not break, we can introduce a new method FileSystem.getNewFileSystem() that skips the CACHE and allocates a new FileSystem object every time it is invoked.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677557#action_12677557 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
+1.
BTW, instead of getUnique(), newInstance() or create() sounds better to me. Also, clarify the semantic difference between the two in javadoc.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Status: Patch Available (was: Open)
Since this patch adds new API (without changing existing APIs), it is safe to pull this into 0.20.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.19.1, 0.19.0, 0.18.2, 0.18.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650309#action_12650309 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
@Doug,
FileSystem implements Closeable, and if a thread obtains a FileSystem object from path.getFileSystem(conf), should it or should it not call fs.close()? Note that the thread has no knowledge whether the object is shared by other threads or not.
As a mater of fact, what I see throughout the current Hadoop code, the practice is not to call fs.close(). For instance, the SequenceFileRecordReader.close() does not actually close the file system. (And if you do put fs.close() there, you are likely fail tests using MiniMRCluster.)
You may argue most applications do not access more than a couple of file systems, and not closing them is not practically harmful. Then what is the reason of making FileSystem implement Closeable in the first place?
Additionally, the bug could also affect single threaded program if individual functions create their own FileSystem objects and close them after usage. For instance,
{code}
static void depthFirstSearch(Path dir, Configuration conf) throws IOException {
FileSystem fs = dir.getFileSystem(conf);
if (fs.isFile(dir)) {
// process file.
return;
}
FileStatus[] subs = fs.listStatus(dir);
for (FileStatus s : subs) {
depthFirstSearch(s.getPath(), conf);
}
fs.close();
}
{code}
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650364#action_12650364 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
The JVM caches URLStreamHandler implementations in an ever-growing table w/o issue. FileSystem implementations don't have much state, nor do they keep connections open, etc. They're lightweight and small in number. I don't yet see this warranting an incompatible change.
Some ways to deal with this that would not involve incompatible changes might be:
- use a WeakHashMap for the cache and close FileSystem instances in a finalize() method.
- remove the cache, and allocate a new FileSystem instance per call to getFileSystem().
The latter would be cleanest. I wonder if it would have much performance impact? Those few cases where it does should be easy to fix.
But overall I still see this as a non-problem for now.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681455#action_12681455 ]
Doug Cutting commented on HADOOP-4655:
--------------------------------------
> Since this patch adds new API (without changing existing APIs), it is safe to pull this into 0.20.
Except we don't add new features to branches, but only to trunk, right?
Also, what's the use case for newInstanceLocal()? I can't think of one and would prefer not to add that method.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Fix For: 0.20.0
>
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt, FileSystemUnique2.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647404#action_12647404 ]
Hong Tang commented on HADOOP-4655:
-----------------------------------
yes
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Hong Tang
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4655) FileSystem.CACHE should be
ref-counted
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4655:
-------------------------------------
Attachment: FileSystemUnique.txt
Changed name of new API to FileSystem.createNewInstance(). Made corresponding changes to libhdfs to expose this API. Added unit tests.
> FileSystem.CACHE should be ref-counted
> --------------------------------------
>
> Key: HADOOP-4655
> URL: https://issues.apache.org/jira/browse/HADOOP-4655
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, fs
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.20.0
> Reporter: Hong Tang
> Assignee: dhruba borthakur
> Attachments: FileSystemUnique.patch, FileSystemUnique.txt
>
>
> FileSystem.CACHE is not ref-counted, and could lead to resource leakage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.