You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2011/08/18 01:18:27 UTC
[jira] [Created] (HADOOP-7551) LocalDirAllocator should incorporate
LocalStorage
LocalDirAllocator should incorporate LocalStorage
-------------------------------------------------
Key: HADOOP-7551
URL: https://issues.apache.org/jira/browse/HADOOP-7551
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 0.20.204.0
Reporter: Eli Collins
The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HADOOP-7551) LocalDirAllocator should incorporate
LocalStorage
Posted by "Eli Collins (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins closed HADOOP-7551.
-------------------------------
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
> Key: HADOOP-7551
> URL: https://issues.apache.org/jira/browse/HADOOP-7551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7551) LocalDirAllocator should
incorporate LocalStorage
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126382#comment-13126382 ]
Ravi Gummadi commented on HADOOP-7551:
--------------------------------------
>> Therefore if there's a disk failure then every call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of all the configured local dirs.
LocalDirAllocator#AllocatorPerContext#confChanged is actually updating the savedLocalDirs everytime conf is changed. So the main concern raised in the above description i.e. every call to confChanged is not resulting into disk checks (except once per configuration change i.e. once per new bad disk).
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
> Key: HADOOP-7551
> URL: https://issues.apache.org/jira/browse/HADOOP-7551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7551) LocalDirAllocator should
incorporate LocalStorage
Posted by "Eli Collins (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins resolved HADOOP-7551.
---------------------------------
Resolution: Won't Fix
Per MAPREDUCE-3011 the if the conf given to the LocalDirAllocator is always updated by callers (eg with input based on LocalStorage), as is currently the case, it doesn't need to be aware. It would be good if LocalDirAllocator didn't need to be given a new conf but combining LocalDirAllocator and LocalStorage is probably too invasive for a stable release. Closing as won't fix.
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
> Key: HADOOP-7551
> URL: https://issues.apache.org/jira/browse/HADOOP-7551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7551) LocalDirAllocator should
incorporate LocalStorage
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126383#comment-13126383 ]
Ravi Gummadi commented on HADOOP-7551:
--------------------------------------
I mean the concern seems to be not valid.
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
> Key: HADOOP-7551
> URL: https://issues.apache.org/jira/browse/HADOOP-7551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7551) LocalDirAllocator should
incorporate LocalStorage
Posted by "Eli Collins (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HADOOP-7551:
-----------------------------------
Assignee: Eli Collins
> LocalDirAllocator should incorporate LocalStorage
> -------------------------------------------------
>
> Key: HADOOP-7551
> URL: https://issues.apache.org/jira/browse/HADOOP-7551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> The o.a.h.fs.LocalDirAllocator is not aware of o.a.h.m.t.LocalStorage (introduced in MAPREDUCE-2413) - it always considers the configured local dirs, not just the ones that happen to be good. Therefore if there's a disk failure then *every* call to get a local path will result in LocalDirAllocator#confChanged doing a disk check of *all* the configured local dirs. It seems like LocalStorage should be a private class to LocalAllocator so that all users of LocalDirAllocator benefit from the disk failure handling and all the various users of LocalDirAllocator don't have to be modified to handle disk failures. Note that LocalDirAllocator already handles faulty directories.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira