You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2008/09/19 18:14:44 UTC
[jira] Updated: (HBASE-851) Region is left unassigned after a
split/rebalancing, throws NSRE
[ https://issues.apache.org/jira/browse/HBASE-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-851:
-------------------------------------
Affects Version/s: 0.2.1
Was on a 0.2.1 cluster
> Region is left unassigned after a split/rebalancing, throws NSRE
> ----------------------------------------------------------------
>
> Key: HBASE-851
> URL: https://issues.apache.org/jira/browse/HBASE-851
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0, 0.2.1
> Reporter: Jean-Daniel Cryans
> Fix For: 0.19.0
>
>
> Master log:
> {code}
> 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:27,174 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server 192.168.1.95:60020 is overloaded. Server load: 8 avg: 7.0
> <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 1 regions. mostLoadedRegions has 8 regions in it.
> <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,174 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:27,175 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 open on 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:27,175 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> <jdcryans> 2008-08-28 12:12:27,175 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row web_pages,http://www.salonskincare.co.uk/product_info.php/products_id/168,1219939934794 in region .META.,,1 with startcode 1219931259154 and server 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:30,352 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:1
> <jdcryans> 2008-08-28 12:12:32,557 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 103, Num Servers: 15, Avg Load: 7.0
> <jdcryans> 2008-08-28 12:12:34,093 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:34,093 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 open on 192.168.1.95:60020
> <jdcryans> 2008-08-28 12:12:34,093 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> <jdcryans> 2008-08-28 12:12:34,093 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794 in region .META.,,1 with startcode 1219931259154 and server 192.168.1.95:60020
> {code}
> HRS 192.168.1.95
> {code}
> jdcryans> 2008-08-28 12:12:24,953 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,307 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794: [B@f0a360
> <jdcryans> 2008-08-28 12:12:27,307 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794: [B@f0a360
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:27,308 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1860667227/attribute
> <jdcryans> 2008-08-28 12:12:27,308 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> 2008-08-28 12:12:34,246 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@552a4a, row => http://www.simplewebengines.com/, {column => attribute:traveliness, value => '...', column => attribute:processed_at, value => '...', column => attribute:content, value => '...', column => attribute:refs, value => '...', column => attribute:crawled_at, value => '...', column => att
> <jdcryans> ribute:html, value => '...', column => attribute:crawled, value => '...'}) from 192.168.1.96:50102: error: org.apache.hadoop.hbase.NotServingRegionException: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> <jdcryans> org.apache.hadoop.hbase.NotServingRegionException: web_pages,http://www.senior-community.net/michigan/charlevoix.htm,1219939934794
> NSRE for a hundred times
> {code}
> Restarting the cluster cleared the issue but this is a nasty bug. Proposed bandaid would be that if we have a NSRE after the retries, asked the master to scan the HRS to see if it's located somewhere else. If not, assign it somewhere. Finally update META.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.