You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2012/06/07 17:01:26 UTC
[jira] [Created] (HADOOP-8490) Add Configuration to FileSystem
cache key
Daryn Sharp created HADOOP-8490:
-----------------------------------
Summary: Add Configuration to FileSystem cache key
Key: HADOOP-8490
URL: https://issues.apache.org/jira/browse/HADOOP-8490
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0-alpha, 0.23.0, 0.24.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291118#comment-13291118 ]
Todd Lipcon commented on HADOOP-8490:
-------------------------------------
bq. In this jira, I'm only proposing that a user can take specific action to ensure a unique yet cached fs instance can be created and later closed.
Couldn't the user just use {{FileSystem.newInstance}}, so that it doesn't use the cache?
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291095#comment-13291095 ]
Daryn Sharp commented on HADOOP-8490:
-------------------------------------
Good points, but I think the issue of sanely managing dynamic vs static values is a separate issue. On that tangent, {{FileSystem}} probably isn't a good place to make that decision since the actual underlying clients of the filesystems are the ones that need to make that determination. It's a much harder problem than this one.
In this jira, I'm only proposing that a user can take specific action to ensure a unique yet cached fs instance can be created and later closed.
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291087#comment-13291087 ]
Kihwal Lee commented on HADOOP-8490:
------------------------------------
Making conf as part of the cache key solves number of problems, but may not be ideal by itself.
We need to think about what are the parts of conf that make sense to be dynamic. Some might need to be allowed to change run-time, others need to stay the same once an input or output stream is created. Creating unique instances for all these variations may not be ideal. Probably FileSystem and the gut should only look at small static part of conf and allow each usage to tweak the behavior in a way that does not interfere with others that share objects/resources.
I believe FileSystem cache was originally intended to work this way. The problem is, it currently doesn't. We need a better way to maintain this separation.
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291187#comment-13291187 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8490:
------------------------------------------------
It sounds like that NM has a specific need on caching. It may make sense to disable FileSystem cache (using fs.hdfs.impl.disable.cache) for NM and then it implements its own cache.
Adding conf as a key to current FileSystem cache may break many existing applications.
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291229#comment-13291229 ]
Daryn Sharp commented on HADOOP-8490:
-------------------------------------
The cache partitioning would be extremely useful when you _do want_ the fs instance cached, but don't want it to collide with other cached instances. Ex. you want to be able to close all the filesystems that have been acquired after an operation. It's not a one-off thing in the NM. It would also be nice for the FsShell to get a partition in the cache to avoid closing a caller's filesystems.
One problem is when code does a {{FileSystem.get(...)}} in many places and expects to get the same fs instance. Using {{newInstance}}, which I originally thought of, does not address that case w/o modifying all the code (which isn't always feasible) to pass around and use that fs instance. Code that does {{Path#getFileSystem(...)}} won't necessarily know up front which unique instances to obtain and then converting the code to pass around and use the list of unique instances would be extremely painful. Basically the user is forced to maintain their own cache of filesystems...
bq. Adding conf as a key to current FileSystem cache may break many existing applications.
How so? The cache key is invisible/private?
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291068#comment-13291068 ]
Daryn Sharp commented on HADOOP-8490:
-------------------------------------
Not honoring the given conf causes the obvious problem of not being able to tweak values.
It's also causing problem for the NM. When an app is done, it should be able to call {{FileSystem.closeAllForUGI}} just like the JT does. Unfortunately that may pull the rug out from under another app for that user also running on the NM. It also means that multiple jobs for the same user are erroneously using the first job's conf. Both are probably latent issues in the JT but go unnoticed or are masked by retries.
Ideally the {{hashCode}} or {{identityHashCode}} would be added to the cache key. A key/value equivalence test should not be performed because seemingly identical confs (ex. cloned from each other) would initially appear the same but may later change. One potential issue is cloned confs that really should be the same -- ex. yarn often creates a {{YarnConfiguration(conf)}}. This won't be a problem if the conversion is done once and stashed. If it's done on the fly multiple times, then it does present a problem. Arguably that would be a bug but it would be difficult to fix in a timely manner.
So an alternative is to add a key to the conf (ex. {{fs.cache-id}}) that can be used in the fs cache key. This would allow partitioning of the cache, albeit imperfectly, that would account for cloned confs that should be treated the same. The onus is placed upon the caller to explicitly change the key when needed, but it would be more transparent for existing code.
I'll wait for comments before preceding.
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8490) Add Configuration to FileSystem
cache key
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395977#comment-13395977 ]
Robert Joseph Evans commented on HADOOP-8490:
---------------------------------------------
I think that it would be nice to actually fix the cache if at all possible. I have heard that most multi-tenet long running systems that use HDFS have to disable the cache and then write their own, i.e. Oozie and Hive. The JobTracker is the only one I know of that does not do this. Most client code I have seen really relies on the fact that getting a "new" FileSystem is cheap, so completely removing the cache is not a feasible option. Look at the MR/Yarn code. Just uploading aggregated log files to HDFS creates about 5 FileSystems if the cache is disabled.
If our users have to disable and work around a "feature" that we cannot turn off we should take that as a bad sign and try to provide a better solution for them. Now if the "fix" makes the performance horrible, or there are other problems we may need to rethink things, but I am +1 on trying to fix the cache.
> Add Configuration to FileSystem cache key
> -----------------------------------------
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given {{Configuration}} into consideration before returning an existing fs instance from the cache with a possibly different conf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira