You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Manjunath Anand (JIRA)" <ji...@apache.org> on 2017/01/22 13:38:27 UTC
[jira] [Commented] (HADOOP-13726) Enforce that FileSystem
initializes only a single instance of the requested FileSystem.
[ https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833516#comment-15833516 ]
Manjunath Anand commented on HADOOP-13726:
------------------------------------------
I saw the new method computeIfAbsent introduced in ConcurrentHashMap as part of Java 8 and felt appropriate for this scenario. Here's the implementation:-
{code}
private final ConcurrentMap<Key, FileSystem> map = new ConcurrentHashMap<>();
private FileSystem getInternal(URI uri, Configuration conf, Key key)
throws IOException{
FileSystem fs;
fs = map.get(key);
if (fs != null) {
return fs;
}
IOException[] ie = new IOException[1];
map.computeIfAbsent(key, k -> {
FileSystem fsc = null;
try {
fsc = createFileSystem(uri, conf);
fsc.key = key;
} catch (IOException e) {
ie[0] = e;
}
return fsc;
});
if(ie[0] != null) throw ie[0];
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return map.get(key);
}
{code}
This avoids synchronization and wont stop all the threads , just those threads referring to the same key. Please let me know your thoughts.
> Enforce that FileSystem initializes only a single instance of the requested FileSystem.
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Reporter: Chris Nauroth
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by multiple call sites or multiple threads. The current implementation does provide this guarantee, but there is a brief race condition window during which multiple threads could perform redundant initialization. If the file system implementation has expensive initialization logic, then this is wasteful. This issue proposes to eliminate that race condition and guarantee initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org