You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michael Bieniosek (JIRA)" <ji...@apache.org> on 2008/09/09 00:29:45 UTC

[jira] Created: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

When dfs restarts or moves blocks around, hbase regionservers don't notice
--------------------------------------------------------------------------

                 Key: HBASE-879
                 URL: https://issues.apache.org/jira/browse/HBASE-879
             Project: Hadoop HBase
          Issue Type: Bug
          Components: regionserver
    Affects Versions: 0.2.0
            Reporter: Michael Bieniosek
            Priority: Critical


Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654288#action_12654288 ] 

Andrew Purtell commented on HBASE-879:
--------------------------------------

Had another of what I first thought was an instance of HBASE-1046.

Some regions were damaged during a write test last night: lease expired exceptions, NotYetReplicated exceptions, "file does not exist" errors on mapfile data files, EOF exceptions while processing reconstruction logs. Compactions/splits failed, leading to dead regions in both of my large tables and complete loss of ability to scan the tables in question.

Enable/disable table did not resolve the problem of dead regions. However, after a full restart of all HBase daemons, all tables are fully back online. This last point is interesting. 

Based on this there should be something that can be done to bring the regions back from the dead without resorting
to a complete shutdown of all HBase daemons, correct? Can the regionservers just reinitialize their DFS client if they start taking filesystem exceptions?

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629537#action_12629537 ] 

Michael Bieniosek commented on HBASE-879:
-----------------------------------------

The regionserver throws an exception like "Can't get block location" or "all datanodes are bad"; this exception is then thrown back to the client.

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643659#action_12643659 ] 

Michael Bieniosek commented on HBASE-879:
-----------------------------------------

HADOOP-3831 is related, but not quite the same.  HBase should be able to survive the block shuffle caused by restarting the dfs entirely.  So I suppose the test for this one should be: restart the dfs, and get the regionservers to deal with blocks moving to different datanodes.

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-879:
---------------------------------

    Affects Version/s:     (was: 0.2.0)
                       0.19.0
                       0.18.1

Updated affected versions. Not just an 0.2.0 issue. 

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.18.1, 0.19.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629357#action_12629357 ] 

Rong-En Fan commented on HBASE-879:
-----------------------------------

Do you see errors from your HBase client or just in region server log?


> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Bieniosek updated HBASE-879:
------------------------------------

    Priority: Major  (was: Critical)

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643550#action_12643550 ] 

stack commented on HBASE-879:
-----------------------------

This was on 0.18.0 hadoop Michael?  I wonder if it HADOOP-3831?  It doesn't happen anymore now our cluster is rebalanced?

> When dfs restarts or moves blocks around, hbase regionservers don't notice
> --------------------------------------------------------------------------
>
>                 Key: HBASE-879
>                 URL: https://issues.apache.org/jira/browse/HBASE-879
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.2.0
>            Reporter: Michael Bieniosek
>
> Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region.  It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.