You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2012/11/29 20:40:58 UTC

[jira] [Created] (HDFS-4239) Means of telling the datanode to stop using a sick disk

stack created HDFS-4239:
---------------------------

             Summary: Means of telling the datanode to stop using a sick disk
                 Key: HDFS-4239
                 URL: https://issues.apache.org/jira/browse/HDFS-4239
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: stack


If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:

1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.

2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.

Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.

Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Harsh J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506731#comment-13506731 ] 

Harsh J commented on HDFS-4239:
-------------------------------

Todd's proposal would still require the DN to be started up with a > 1 toleration value via {{dfs.datanode.failed.volumes.tolerated}}.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Tibor Vass (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510155#comment-13510155 ] 

Tibor Vass commented on HDFS-4239:
----------------------------------

What about stopping the datanode, chmod 0-ing, and restarting the datanode?
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Harsh J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506734#comment-13506734 ] 

Harsh J commented on HDFS-4239:
-------------------------------

Typo, I meant >= 1
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510939#comment-13510939 ] 

Jean-Daniel Cryans commented on HDFS-4239:
------------------------------------------

bq. That should work just fine, if the HDFS config is compatible with the new set of available directories. 

We tried it today, it worked fine. We did encounter an interesting problem tho, the region server on the same node continued to use that disk directly since it's configured with local reads.

To rephrase that, a long running BlockReaderLocal will ride over local DN restarts and disk "ejections". We had to drain the RS of all its regions in order to stop it from using the bad disk.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510181#comment-13510181 ] 

Andy Isaacson commented on HDFS-4239:
-------------------------------------

bq. What about stopping the datanode, chmod 0-ing, and restarting the datanode?

That should work just fine, if the HDFS config is compatible with the new set of available directories.  That means either ensuring that the inaccessible datadir does not exceed the {{dfs.datanode.failed.volumes.tolerated}} value, or removing the inaccessible datadir from {{dfs.data.dir}}.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506791#comment-13506791 ] 

Andy Isaacson commented on HDFS-4239:
-------------------------------------

To expand on my previous comment:

I tested on trunk, on a DN with {{dfs.datanode.failed.volumes.tolerated=1}}.  Running {{chmod 0 /data/5/datadir/current}} caused the DN to eject the volume and continue operating.  I then used {{lsof -p}} to verify what filedescriptors remained open and observed that {{/data/5/datadir/in_use.lock}} was still open.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506719#comment-13506719 ] 

Todd Lipcon commented on HDFS-4239:
-----------------------------------

Would {{chmod 000 /path/to/data/dir}} do the trick? That should cause it to start getting IOExceptions, which would then get it to eject that disk from its list.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506758#comment-13506758 ] 

Andy Isaacson commented on HDFS-4239:
-------------------------------------

bq. would a umount -f let you force the unmount. 

Unfortunately not while a running process still has a filedescriptor open on the volume. It would require {{revoke()}} support in the kernel, and that effort foundered many years ago.  http://lwn.net/Articles/262528/
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506747#comment-13506747 ] 

Steve Loughran commented on HDFS-4239:
--------------------------------------

would a {{umount -f}} let you force the unmount. 

The big volume management JIRA is HDFS-1362 -that's the one that really needs finishing off. 
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506769#comment-13506769 ] 

Andy Isaacson commented on HDFS-4239:
-------------------------------------

bq. Would chmod 000 /path/to/data/dir do the trick? That should cause it to start getting IOExceptions, which would then get it to eject that disk from its list.

Unfortunately the DN keeps the {{in_use.lock}} open even after the volume is marked failed.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506724#comment-13506724 ] 

stack commented on HDFS-4239:
-----------------------------

Let me try.
                
> Means of telling the datanode to stop using a sick disk
> -------------------------------------------------------
>
>                 Key: HDFS-4239
>                 URL: https://issues.apache.org/jira/browse/HDFS-4239
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: stack
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode.  If the datanode is carrying 6 or 12 disks of data, especially on a cluster that is smallish -- 5 to 20 nodes -- the rereplication of the downed datanode's data can be pretty disruptive, especially if the cluster is doing low latency serving: e.g. hosting an hbase cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You can't unmount the disk while it is in use).  This latter is better in that only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop using a disk an operator has designated 'bad'.  This would be like option #2 above minus the need to stop and restart the datanode.  Ideally the disk would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk after its been replaced.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira