You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jinglun (JIRA)" <ji...@apache.org> on 2019/08/08 13:32:00 UTC
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close
child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jinglun updated HADOOP-15565:
-----------------------------
Description:
ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. It's children filesystems are cached in FileSystem.CACHE and shared by all the ViewFileSystem instances. We could't simply close all the children filesystems because it will break the semantic of FileSystem.newInstance().
We might add an inner cache to ViewFileSystem, let it cache all the children filesystems. The children filesystems are not shared any more. When ViewFileSystem is closed we close all the children filesystems in the inner cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't be too many FileSystem instances.
The FileSystem.CACHE caches the ViewFileSysem instance and the other instances(the children filesystems) are cached in the inner cache.
was:
When we create a ViewFileSystem, all it's child filesystems will be cached by FileSystem.CACHE. Unless we close these child filesystems, they will stay in FileSystem.CACHE forever.
I think we should let FileSystem.CACHE cache ViewFileSystem only, and let ViewFileSystem cache all it's child filesystems. So we can close ViewFileSystem without leak and won't affect other ViewFileSystems.
I find this problem because i need to re-login my kerberos and renew ViewFileSystem periodically. Because FileSystem.CACHE.Key is based on UserGroupInformation, which changes everytime i re-login, I can't use the cached child filesystems when i new a ViewFileSystem. And because ViewFileSystem.close does nothing but remove itself from cache, i leak all it's child filesystems in cache.
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. It's children filesystems are cached in FileSystem.CACHE and shared by all the ViewFileSystem instances. We could't simply close all the children filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children filesystems. The children filesystems are not shared any more. When ViewFileSystem is closed we close all the children filesystems in the inner cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org