You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Gabor Bota (JIRA)" <ji...@apache.org> on 2019/08/05 11:22:00 UTC
[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't
close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900007#comment-16900007 ]
Gabor Bota edited comment on HADOOP-15565 at 8/5/19 11:21 AM:
--------------------------------------------------------------
Thanks for working on this [~LiJinglun]!
{quote}I think we should let FileSystem.CACHE cache ViewFileSystem only, and let ViewFileSystem cache all it's child filesystems.{quote}
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of {{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances right? If you mean that (based on the code you do) please correct the description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with {quote}re-login my kerberos and renew ViewFileSystem periodically{quote}. Could you write a test case where this is reproduced to show that this change will solve that issue?
was (Author: gabor.bota):
Thanks for working on this [~LiJinglun]!
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let ViewFileSystem cache all it's child filesystems.
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of {{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances right? If you mean that (based on the code you do) please correct the description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with "re-login my kerberos and renew ViewFileSystem periodically". Could you write a test case where this is reproduced to show that this change will solve that issue?
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch
>
>
> When we create a ViewFileSystem, all it's child filesystems will be cached by FileSystem.CACHE. Unless we close these child filesystems, they will stay in FileSystem.CACHE forever.
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let ViewFileSystem cache all it's child filesystems. So we can close ViewFileSystem without leak and won't affect other ViewFileSystems.
> I find this problem because i need to re-login my kerberos and renew ViewFileSystem periodically. Because FileSystem.CACHE.Key is based on UserGroupInformation, which changes everytime i re-login, I can't use the cached child filesystems when i new a ViewFileSystem. And because ViewFileSystem.close does nothing but remove itself from cache, i leak all it's child filesystems in cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org