You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/07/25 23:03:31 UTC
[jira] Created: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Master not reassigning .META. from failed/failing regionserver
--------------------------------------------------------------
Key: HBASE-776
URL: https://issues.apache.org/jira/browse/HBASE-776
Project: Hadoop HBase
Issue Type: Bug
Components: master
Affects Versions: 0.2.0
Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
Reporter: Andrew Purtell
Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617062#action_12617062 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
There is no indication in the regionserver log why RPC becomes unresponsive.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-776:
---------------------------------
Attachment: hbase-hadoop-regionserver-sjdc-atr-dc-13.log
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-776.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.2.0
HBASE-751 is supposed to contain fix. Will reopen if seen again.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Fix For: 0.2.0
>
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617116#action_12617116 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
Just saw a period where a new steady state was reached with 1800-2000 requests for META per second.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617115#action_12617115 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
Is 200-500 requests for META per second is normal? The client code is reusing the same HTable object and is just using table.commit(BatchUpdate) to insert data imported from a Postgres DB.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617124#action_12617124 ]
stack commented on HBASE-776:
-----------------------------
Andrew, can you run with DEBUG level logging -- we might get a clue why it goes 'deaf' -- and can you up your ulimit for file descriptors from 1024 (because out-of-FDs can manifest in weird ways)? See FAQ for how.
A socket timeout when scanning meta should provoke relocation of meta I'd say. Need to dig more. From your logs, it happens over and over and the master just sits there dumb.
Regards pounding META, you'd think the META info cached. Let me do loading here and log the META region accesses. How many clients you running?
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617127#action_12617127 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
Ok, I'll up ulimit to 10K for the hadoop account and see what happens. Was already planning to sit down and try to produce a simple program that reproduces the problem reliably. If I can get something consistent I'll run in DEBUG and if it still triggers I'll upload the logs. Maybe I'll find the client is doing something dumb. However as I can see it is just one client with two threads doing nothing particularly odd or special.
The cluster configuration is 15 nodes:
1 master/regionserver/namenode/datanode/jobtracker
14 regionserver/datanode/tasktracker
Each node is dual processor 4 core Xeons with 8GB RAM.
The master node does not load up.
Client is external to the cluster.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617375#action_12617375 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
The excessive queries to META was a client issue. Sorry for the noise. There was an existence test buried in some old code which created HBaseAdmin objects. It did apparently serve as a torture test, though.
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-776:
---------------------------------
Attachment: master_gui.png
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-776:
---------------------------------
Attachment: hbase-hadoop-master-sjdc-atr-dc-1.log
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617064#action_12617064 ]
Andrew Purtell commented on HBASE-776:
--------------------------------------
During the duration of this incident the regionserver in question was being pounded while the rest of the cluster was idle. Wouldn't it make sense to migrate META from a heavily loaded regionserver if another is substantially less loaded in general?
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617212#action_12617212 ]
stack commented on HBASE-776:
-----------------------------
Its looking like this is symptom of HBASE-751
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-776) Master not reassigning .META. from
failed/failing regionserver
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617208#action_12617208 ]
stack commented on HBASE-776:
-----------------------------
This exception of yours Andrew seems pretty easy to manufacture. I see it here in a little test I'm running. Kept getting the exception over and over for 30 mins now.
{code}
2008-07-26 19:09:22,111 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan one META region: {regionname: .META.,,1, startKey: <>, server: XX.XX.XX.XX:60020}
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:559)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
at $Proxy2.openScanner(Unknown Source)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:159)
at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:69)
at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:124)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:139)
at org.apache.hadoop.hbase.Chore.run(Chore.java:63)
{code}
> Master not reassigning .META. from failed/failing regionserver
> --------------------------------------------------------------
>
> Key: HBASE-776
> URL: https://issues.apache.org/jira/browse/HBASE-776
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Environment: CentOS x86_64, JDK 1.6, Hadoop 0.17.1, HBase 0.2.0, r679585, Fri Jul 25 16:47:26 UTC 2008
> Reporter: Andrew Purtell
> Attachments: hbase-hadoop-master-sjdc-atr-dc-1.log, hbase-hadoop-regionserver-sjdc-atr-dc-13.log, master_gui.png
>
>
> In our environment sometimes the regionserver carrying META is also assigned to the 'content' table, into which objects retrieved from Internet crawling is stored. For unclear reason the regionserver occasionally goes "deaf" (seperate issue) and when this happens META is no longer available. The master then never reassigns META, so the whole cluster is down from this point and does not recover. Logs attached.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.