You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Boris Shkolnik (JIRA)" <ji...@apache.org> on 2009/06/30 20:58:47 UTC
[jira] Created: (HDFS-457) better handling of volume failure in
Data Node storage
better handling of volume failure in Data Node storage
------------------------------------------------------
Key: HDFS-457
URL: https://issues.apache.org/jira/browse/HDFS-457
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Reporter: Boris Shkolnik
Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883423#action_12883423 ]
Jeff Zhang commented on HDFS-457:
---------------------------------
I noticed that only the BlockSender will do checkDisk, so only read operation will make the namenode know that one disk is failed. Has anyone try this patch ? I doubt there will still be some problems with this patch.
In the testcase TestDataNodeVolumeFailure, I make a little changes, replace the triggerFailure with a write operation (in this case the datanode do not know one volume is failed, and then the only one replica will be write successfully, and waiting for another replica copied from another datanode for a long time.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457-1.patch
fixed hudson errors
and implemented review suggestions.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740987#action_12740987 ]
Edward Capriolo commented on HDFS-457:
--------------------------------------
Also , I just ran into this . One of my disks failed however it was available on the system as read only. Some of the directories produce io errors when read. Will this patch be able to deal with that condition?
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-457:
----------------------------------------
Attachment: TestFsck.zip
TestFsck.zip: downloaded the log file from the build and extracted TestFsck log.
The log is unexpectedly long, which has 34MB. Could you take a look, Boris?
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HDFS-457:
---------------------------------
Release Note: Datanode can continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed. (was: Do not shutdown datanode if some, but not all, volumes fail.)
Editorial pass over all release notes prior to publication of 0.21.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739773#action_12739773 ]
Tsz Wo (Nicholas), SZE commented on HDFS-457:
---------------------------------------------
- There is a second version of LOG.warn(..) which accepts an Exception object as a parameter. You may consider changing
{code}
+ DataNode.LOG.warn("IOException in BlockReceiver constructor. Cause is " +
+ cause);
{code}
to
{code}
+ DataNode.LOG.warn("IOException in BlockReceiver constructor.", cause);
{code}
and the other LOG.warn/info/err calls.
- DataNode.checkDiskError() should not be public.
- The return type of checkDirs() shuold be List<FSVolume>.
- Looks like that removed_vols.size() is equal to removed, or rmoved_vols == null when removed == 0. So we may drop removed.
- We may omit toSting() calls in the following:
{code}
+ DataNode.LOG.warn("Removing failed volume " + fsv.toString() + " - " + e.getLocalizedMessage());
{code}
{code}
+ "volumes. List of current volumes: " + toString());
{code}
- It is hard to related the name "keepRunning" with "checks how many valid storage volumes are there in the DataNode". How about change it to something like "hasEnoughVolumes", "hasEnoughResource" or "isSystemHealthy?
- The second "f = null" is redundant since f == null if validateBlockFile throws an exception.
{code}
+ File f = null;;
+ try {
+ f = validateBlockFile(b);
+ } catch(IOException e) {
+ f = null;
+ }
{code}
I suggest to print some messages in the catch block.
- checkDataDir() changes volumeMap but is not synchronized on it.
- Also, would the new checkDataDir() implementation take a long time to execute? We may need some performance test on this.
- It is better not to change the existing constant values.
{code}
- final static int DISK_ERROR = 1;
- final static int INVALID_BLOCK = 2;
+ final static int DISK_ERROR = 1; // there are still valid volumes on DN
+ final static int FATAL_DISK_ERROR = 2; // no valid volumes left on DN
+ final static int INVALID_BLOCK = 3;
{code}
How about set FATAL_DISK_ERROR to 3?
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743613#action_12743613 ]
Hadoop QA commented on HDFS-457:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12416633/HDFS-457-2.patch
against trunk revision 804127.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/69/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/69/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/69/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/69/console
This message is automatically generated.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740852#action_12740852 ]
Hadoop QA commented on HDFS-457:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12415899/HDFS-457-1.patch
against trunk revision 802264.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/50/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/50/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/50/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/50/console
This message is automatically generated.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743015#action_12743015 ]
Hadoop QA commented on HDFS-457:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12416492/HDFS-457-2.patch
against trunk revision 803973.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/66/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/66/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/66/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/66/console
This message is automatically generated.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Patch Available (was: Open)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741435#action_12741435 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
Disk is checked by DiskChecker.checkDir(). It checks mkdir, read and write. So in this case it should consider the volume as failed (because it cannot write to it).
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457-2.patch
fixed two issues:
1. make sure that blockreport is sent before starting verification
2. delete files in the directory instead of the whole directory (because deleting
directory with locked files in it may fail on Windows or on NFS).
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-457:
-------------------------------------
Attachment: HDFS-457-y20.patch
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457-y20.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Uma Mahesh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886751#action_12886751 ]
Uma Mahesh commented on HDFS-457:
---------------------------------
Hi,
checkDirs will throw null pointer exception.
ArrayList<FSVolume> removed_vols = null;
........
........catch(.....){
removed_vols = new ArrayList();
}
..........
removed_vols .size ()
if no exception then above line will throqw null pointer exception.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457.patch
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883466#action_12883466 ]
Jeff Zhang commented on HDFS-457:
---------------------------------
Create Jira HDFS-1273 for this issue, and put patch there
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725747#action_12725747 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
We should try to keep DataNode alive even if one of its volumes is not accessible. When DataNode handles the error we can check to see what percentage of the volumes is down. If it more then some predefined threshold we should shut the node down. But if it is not - we can keep it alive. In this case we need to do the following:
o remove the volume from the list of valid volumes
o go over all the blocks and remove those that reside on this volume
o immediately schedule a block report to update namenode and start replication.
o Optional. We can try to monitor removed volumes (or periodically compare valid ones against the configured ones), and if some of them comes back to live (or on an operator command) we may try to restore it. (I don't know if it is possible and plausible, but it can be designed/done as the next step).
Affected classes and methods:
* BlockReceiver constructor
* BlockReceiver run()
* BlockReceiver lastDataNodeRun()
* FSDataset, FSVolume, FSVolumeSet, FSDir
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744463#action_12744463 ]
Hudson commented on HDFS-457:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #53 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/53/])
. Do not shutdown datanode if some, but not all, volumns fail. Contributed by Boris Shkolnik
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744214#action_12744214 ]
Mahadev konar commented on HDFS-457:
------------------------------------
i would suggest fixing the spelling for shutdown as well :)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726044#action_12726044 ]
dhruba borthakur commented on HDFS-457:
---------------------------------------
Another big issue is "reporting". In the current code, an adminstrator sees the dead Datanode in the webui and knows that he/she has to fix it.
In your proposal, a datanode could stop using a portion of its disk and it would be very worthwhile to report this to the adminstrator. Otherwise, a certain percentage of disk space could leak out of HDFS usage without anybody knowing about it.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Patch Available (was: Open)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HDFS-457:
-------------------------------------
Affects Version/s: 0.20-append
Should add to 0.20-append for durability
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.20-append
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Open (was: Patch Available)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457-2.patch
re-uploading the patch to get it run by hudson.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457-3.patch
implemented comments by Nicholas.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Open (was: Patch Available)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744189#action_12744189 ]
Tsz Wo (Nicholas), SZE commented on HDFS-457:
---------------------------------------------
A few nits:
- change numberOfVolumes() to private
- keep handleDiskError(String errMsgr) private
- "shutdonw" should be "shutdown"
{code}
+ // to shutdonw DN completely and don't want NN to remove it.
{code}
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744227#action_12744227 ]
Mahadev konar commented on HDFS-457:
------------------------------------
sorry ignore my comment, didnt read nicholas's comments...
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Attachment: HDFS-457-2.patch
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732695#action_12732695 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
we should also handle failed reads from the failed volumes.
we can take care of it in FSDataset.validateBlockFile() because it is called on every "read" operation.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HDFS-457:
-------------------------------------
Attachment: HDFS-457_20-append.patch
New patch should apply cleanly to 0.20-append. Also imported the VolumeFailure unit test
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.20-append
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-457:
----------------------------------------
Fix Version/s: 0.21.0
Issue Type: Improvement (was: Bug)
Hadoop Flags: [Reviewed]
+1 patch looks good.
The diff between HDFS-457-3.patch and HDFS-457-2.patch is minor. So I think we don't have to submit it to Hudson again for saving some cycles.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743313#action_12743313 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
the failed test, testFsckMove, has timed out.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732693#action_12732693 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
good point.
I will report to NN using errorReport interface. Currently if NN received DatanodeProtocol.DiskError message it removes the datanode.
I will introduce a DatanodeProtocol.FatalDiskError - this should remove the datanode. But DiskError will just cause NN to log a WARN message about failed volume.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Erik Steffl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Steffl updated HDFS-457:
-----------------------------
Attachment: jira.HDFS-457.branch-0.20-internal.patch
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883447#action_12883447 ]
Jeff Zhang commented on HDFS-457:
---------------------------------
This is my first patch on HDFS, not sure whether it is right to attach the patch here. or do need to create a new jira for this issue ?
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik reassigned HDFS-457:
-----------------------------------
Assignee: Boris Shkolnik
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740680#action_12740680 ]
Boris Shkolnik commented on HDFS-457:
-------------------------------------
thanks for the comments.
>> * Also, would the new checkDataDir() implementation take a long time to execute? We may need some performance test on this.).
This function makes one loop over ALL the blocks. I believe this is what causes the concern. I've looked at it and ran some tests.
1. This loop will happen only in case when there is an actual failure. I think it is expected to have some extra work to be done in this case.
2. This extra work shouldn't be so bad. I've ran test for 1500 blocks it took about 600mls, for 5000 about 800mls. For huge nodes with hundreds of thousands blocks it still should be tolerable.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-457:
----------------------------------------
Resolution: Fixed
Release Note: Do not shutdown datanode if some, but not all, volumes fail.
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Boris!
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883456#action_12883456 ]
Eli Collins commented on HDFS-457:
----------------------------------
Hey Jeff,
Nice catch. Please file a new jira.
Thanks,
Eli
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Patch Available (was: Open)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732789#action_12732789 ]
Hadoop QA commented on HDFS-457:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12413868/HDFS-457.patch
against trunk revision 794953.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
-1 release audit. The applied patch generated 285 release audit warnings (more than the trunk's current 282 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/23/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/23/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/23/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/23/console
This message is automatically generated.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-457:
-----------------------------
Affects Version/s: (was: 0.20-append)
Removing 0.20-append tag - this isn't append-specific.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated HDFS-457:
----------------------------
Attachment: HDFS_457.patch
Attach patch. Do checkDiskError in BlockReceiver before allocating volumes for the new block. So after checkDiskError, it is guaranteed that the volumes are all normal, the failed volumes has been removed.
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Fix For: 0.21.0
>
> Attachments: HDFS-457-1.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-2.patch, HDFS-457-3.patch, HDFS-457.patch, HDFS-457_20-append.patch, HDFS_457.patch, jira.HDFS-457.branch-0.20-internal.patch, TestFsck.zip
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Patch Available (was: Open)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-457) better handling of volume failure in
Data Node storage
Posted by "Boris Shkolnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Shkolnik updated HDFS-457:
--------------------------------
Status: Open (was: Patch Available)
> better handling of volume failure in Data Node storage
> ------------------------------------------------------
>
> Key: HDFS-457
> URL: https://issues.apache.org/jira/browse/HDFS-457
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Boris Shkolnik
> Assignee: Boris Shkolnik
> Attachments: HDFS-457-1.patch, HDFS-457.patch
>
>
> Current implementation shuts DataNode down completely when one of the configured volumes of the storage fails.
> This is rather wasteful behavior because it decreases utilization (good storage becomes unavailable) and imposes extra load on the system (replication of the blocks from the good volumes). These problems will become even more prominent when we move to mixed (heterogeneous) clusters with many more volumes per Data Node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.