[jira] Updated: (HBASE-555) Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments because open processing is done in series

["stack (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-555:
------------------------

    Attachment: 555-0.1.patch

{code}
M  src/java/org/apache/hadoop/hbase/HRegionServer.java
   (housekeeping): Call a housekeeping method before we go into
   hibernation.  Currently its only task is review of the todo
   list and adding MSG_REGION_PROCESS_OPEN if any regions waiting
   to be opened in the queue.
   (addProcessMessage): Method to add MSG_REPORT_PROCESS_OPEN to
   messages to send the server.
{code}

> Only one Worker in HRS; on startup, if assigned tens of regions, havoc of reassignments because open processing is done in series
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-555
>                 URL: https://issues.apache.org/jira/browse/HBASE-555
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.16.0, 0.2.0, 0.1.0
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>         Attachments: 555-0.1.patch
>
>
> On the Lars clusters, he's up into the thousands of regions.  Starting this cluster, there is a load of churn in the master log as we assign regions, they report their opening and then after the hbase.hbasemaster.maxregionopen of one minute elapses, we assign the region elsewhere.
> Problem seems to be the fact that we only run a single Worker thread in our regionserver; means that region opens are processed in series.
> For example, the below shows when a master assigned a region and then the regionserver side log when it got around to opening it:
> {code}
> 2008-03-29 04:48:51,638 INFO org.apache.hadoop.hbase.HMaster: assigning region pdc-docs,US20060158177_20060720,1205765009844 to server 192.168.105.19:60020
> ..
> 2008-03-29 04:50:58,124 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : pdc-docs,US20060158177_20060720,1205765009844
> {code}
> There is > 2 minutes between the two loggings (I checked clocks on this cluster and they are synced).
> Looking in the regionserver log, its just filled with logging on the opening of regions.  The region opens are running pretty fast at about a second each but there are hundreds of regions to open in this case so its easy to go over our default of 60 seconds.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.