You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andrea <an...@gmail.com> on 2013/12/04 12:18:56 UTC
Re: HBase and HDFS replication
I'm sorry for the delay but in the past days I had troubles with my amazon
account! Anyway, I have this situation (status 'simple' output) with an hdfs
replication set at 3:
3 live servers
ip-10-254-104-164:60020 1386152974983
requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=55,
maxHeapMB=6483
ip-10-232-23-30:60020 1386152980000
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=69,
maxHeapMB=6483
ip-10-253-6-188:60020 1386152977755
requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=1084,
maxHeapMB=6483
0 dead servers
Aggregate load: 0, regions: 8
/////////////////////////////////////////////////////////////////////////
I kill the node ip-10-254-104-164 (the first one) and here the situation:
2 live servers
ip-10-232-23-30:60020 1386152980000
requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=39,
maxHeapMB=6483
ip-10-253-6-188:60020 1386152977755
requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=1087,
maxHeapMB=6483
1 dead servers
ip-10-254-104-164,60020,1386152974983
Aggregate load: 0, regions: 6
/////////////////////////////////////////////////////////////////////////
Note that at now (after more than 30 minutes it doesn't recover the
regions!!!)
And here the master output (the first line is the master adding the killed
node to the dead server):
2013-12-04 10:44:01,907 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Added=ip-10-254-104-164,60020,1386152974983 to dead servers, submitted
shutdown handler to be executed, root=false, meta=false
2013-12-04 10:44:01,907 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
for ip-10-254-104-164,60020,1386152974983
2013-12-04 10:44:01,914 DEBUG
org.apache.hadoop.hbase.master.MasterFileSystem: Renamed region directory:
hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting
2013-12-04 10:44:01,914 INFO org.apache.hadoop.hbase.master.SplitLogManager:
dead splitlog workers [ip-10-254-104-164,60020,1386152974983]
2013-12-04 10:44:01,916 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: Scheduling batch of logs to
split
2013-12-04 10:44:01,918 INFO org.apache.hadoop.hbase.master.SplitLogManager:
started splitting logs in [hdfs://master1:54310/hbase/.logs/ip-10-254-104-
164,60020,1386152974983-splitting]
2013-12-04 10:44:01,926 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at
znode /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:01,928 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 ver = 0
2013-12-04 10:44:01,935 INFO org.apache.hadoop.hbase.master.SplitLogManager:
task /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 acquired by ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:44:02,403 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned =
0
2013-12-04 10:44:03,401 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned =
0
2013-12-04 10:44:04,401 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned =
0
2013-12-04 10:44:04,980 INFO org.apache.hadoop.hbase.master.SplitLogManager:
task /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 entered state done ip-10-253-
6-188,60020,1386152977755
2013-12-04 10:44:04,997 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed
log hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting/ip-10-254-104-164%2C60020%2C1386152974983.1386152986541 to
hdfs://master1:54310/hbase/.oldlogs/ip-10-254-104-
164%2C60020%2C1386152974983.1386152986541
2013-12-04 10:44:04,999 INFO org.apache.hadoop.hbase.master.SplitLogManager:
Done splitting
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:05,003 INFO org.apache.hadoop.hbase.master.SplitLogManager:
finished splitting (more than or equal to) 0 bytes in 1 log files in
[hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting] in 3085ms
2013-12-04 10:44:05,004 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:05,006 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-12-04 10:44:05,006 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2013-12-04 10:44:05,013 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192,}
2013-12-04 10:44:05,016 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined and
split region usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.;
checking daughter presence
2013-12-04 10:44:05,019 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key
'usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98.'
2013-12-04 10:44:05,019 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at
'usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98.'
2013-12-04 10:44:05,023 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98. present
2013-12-04 10:44:05,023 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key
'usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d36
745a.'
2013-12-04 10:44:05,023 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at
'usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d36
745a.'
2013-12-04 10:44:05,027 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d367
45a. present
2013-12-04 10:44:05,027 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined and
split region
usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d367
45a.; checking daughter presence
2013-12-04 10:44:05,029 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key
'usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6
eb0a.'
2013-12-04 10:44:05,029 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at
'usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6
eb0a.'
2013-12-04 10:44:05,033 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. present
2013-12-04 10:44:05,033 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key
'usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c35
2dd1a.'
2013-12-04 10:44:05,033 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at
'usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c35
2dd1a.'
2013-12-04 10:44:05,036 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. present
2013-12-04 10:44:05,037 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:44:05,042 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one;
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2,
available=2) available servers
2013-12-04 10:44:05,042 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:44:05,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:44:05,057 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:05,068 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one;
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2,
available=2) available servers
2013-12-04 10:44:05,068 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:44:05,079 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:05,081 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: 2 regions
which were planned to open on ip-10-254-104-164,60020,1386152974983 have
been re-assigned.
2013-12-04 10:44:05,081 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
processing of shutdown of ip-10-254-104-164,60020,1386152974983
2013-12-04 10:44:05,099 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:05,116 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:11,771 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:11,786 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:15,109 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:15,124 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:18,447 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:18,463 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:21,784 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:21,800 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:25,122 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:25,138 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:28,467 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:28,476 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:31,805 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:31,813 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:35,141 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:35,151 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:38,485 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:38,490 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:41,824 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:41,829 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:45,162 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:45,166 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:48,501 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:48,506 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:51,841 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:51,845 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OPENING, ts=1386153891924,
server=ip-10-232-23-30,60020,1386152980000,
797ead45993e04d649da99d4c352dd1a=usertable,user41007679046964...
2013-12-04 10:44:53,748 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row= for max=2147483647 rows using
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@
4bc900b6
2013-12-04 10:44:53,783 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Deleting region usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.
because daughter splits no longer hold references
2013-12-04 10:44:53,784 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
ARCHIVING region
hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425
2013-12-04 10:44:53,784 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Have an archive directory, preparing to move files
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Archiving:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily]
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Starting to archive files:[class
org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily]
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
moving files to the archive directory:
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25
2013-12-04 10:44:53,793 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Created archive
directory:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d
23ac22a77425
2013-12-04 10:44:53,793 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily
2013-12-04 10:44:53,794 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily is a directory, archiving children files
2013-12-04 10:44:53,796 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
moving files to the archive directory:
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25/family
2013-12-04 10:44:53,800 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Created archive
directory:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d
23ac22a77425/family
2013-12-04 10:44:53,800 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily/8331259e07d7457ab61961cd629ea1f8
2013-12-04 10:44:53,802 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
No existing file in archive
for:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22
a77425/family/8331259e07d7457ab61961cd629ea1f8, free to archive original
file.
2013-12-04 10:44:53,805 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Finished archiving file from: class
org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily/8331259e07d7457ab61961cd629ea1f8, to:
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25/family/8331259e07d7457ab61961cd629ea1f8
2013-12-04 10:44:53,805 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Successfully resolved and archived, now can just delete region.
2013-12-04 10:44:53,808 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver:
Deleted all region files in:
hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425
2013-12-04 10:44:55,179 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:55,188 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:58,517 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:58,526 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:01,858 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:01,865 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:05,194 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:05,202 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:08,533 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:08,540 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:11,871 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:11,878 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:15,209 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:15,216 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:18,547 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:18,554 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:21,885 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:21,899 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:25,224 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:25,237 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:28,562 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:28,575 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:31,900 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:31,913 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:35,238 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:35,251 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:38,576 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:38,590 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:41,914 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:41,928 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,257 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-232-23-
30,60020,1386152980000, region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,258 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,258 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one;
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2,
available=1) available servers
2013-12-04 10:45:45,259 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,259 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a. state=CLOSED, ts=1386153945345, server=ip-10-232-23-
30,60020,1386152980000
2013-12-04 10:45:45,259 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:45:45,265 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,266 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,266 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a.;
plan=hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d8
22f5c0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,266 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,279 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,280 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,280 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one;
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2,
available=1) available servers
2013-12-04 10:45:45,280 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,281 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386153945334, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:45:45,281 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:45:45,283 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,289 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,289 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,289 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.;
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,289 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,303 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,333 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,336 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:52,004 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:52,008 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:55,342 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:55,345 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:58,680 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:58,686 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:02,019 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:02,024 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:05,358 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:05,361 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:08,704 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:08,707 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:12,043 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:12,046 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:15,387 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:15,389 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:18,725 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:18,728 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:22,064 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:22,067 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:25,406 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:25,407 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:28,742 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:28,746 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:32,081 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:32,090 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:35,420 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:35,428 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:38,758 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:38,765 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:42,095 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:42,103 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:45,366 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Failed all from region=.META.,,1.1028785192, hostname=ip-10-232-23-30,
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to ip-10-232-23-30/10.232.23.30:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34262
remote=ip-10-232-23-30/10.232.23.30:60020]
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchCallback(HConnectionManager.java:1711)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1563)
at org.apache.hadoop.hbase.client.HTable.delete(HTable.java:801)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:146)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:133)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteRegion(MetaEditor.java:310)
at
org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.jav
a:242)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:145)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34262 remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
at com.sun.proxy.$Proxy11.multi(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1540)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1538)
at
org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.
java:228)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1547)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1535)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
... 1 more
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34262
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:46:45,371 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Removed .META.,,1.1028785192 for tableName=.META. from cache because of
usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.
2013-12-04 10:46:45,372 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Retry 1, sleep for 1009ms!
2013-12-04 10:46:45,434 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:45,443 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:46,383 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:46:46,386 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is ip-10-232-23-30:60020
2013-12-04 10:46:48,773 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:48,780 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:52,111 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:52,186 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:55,449 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:55,523 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:58,787 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:58,869 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:02,125 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:02,208 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:05,464 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:05,546 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:08,802 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:08,885 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:12,140 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:12,223 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:15,478 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:15,561 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:18,816 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:18,899 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:22,154 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:22,237 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,505 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,505 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,505 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one;
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2,
available=1) available servers
2013-12-04 10:47:25,505 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,505 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a. state=CLOSED, ts=1386154045562, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:47:25,505 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:47:25,514 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,514 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,514 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a.;
plan=hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d8
22f5c0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,514 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,579 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-232-23-
30,60020,1386152980000, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,580 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,580 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one;
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2,
available=1) available servers
2013-12-04 10:47:25,580 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,580 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386154045672, server=ip-10-232-23-
30,60020,1386152980000
2013-12-04 10:47:25,580 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:47:25,587 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,587 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,587 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.;
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,587 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,596 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,628 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:28,968 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:32,306 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:35,644 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:38,982 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:42,320 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:45,657 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:46,438 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Failed all from region=.META.,,1.1028785192, hostname=ip-10-232-23-30,
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to ip-10-232-23-30/10.232.23.30:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchCallback(HConnectionManager.java:1711)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1563)
at org.apache.hadoop.hbase.client.HTable.delete(HTable.java:801)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:146)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:133)
at
org.apache.hadoop.hbase.catalog.MetaEditor.deleteRegion(MetaEditor.java:310)
at
org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.jav
a:242)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:145)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
at com.sun.proxy.$Proxy11.multi(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1540)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1538)
at
org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.
java:228)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1547)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1535)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
... 1 more
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:47:46,440 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Removed .META.,,1.1028785192 for tableName=.META. from cache because of
usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.
2013-12-04 10:47:46,440 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Retry 2, sleep for 1003ms!
2013-12-04 10:47:47,446 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:48,996 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:52,333 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:55,671 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:59,010 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:02,359 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:05,697 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:09,035 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:12,373 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:15,710 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:19,048 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:22,388 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:25,726 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:29,064 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:32,403 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:35,742 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:39,081 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:42,418 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:45,756 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:46,454 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Call openRegion() to ip-
10-232-23-30,60020,1386152980000 has timed out when trying to assign
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a., but the region might already be opened on ip-10-232-23-
30,60020,1386152980000.
java.net.SocketTimeoutException: Call to ip-10-232-23-30/10.232.23.30:60020
failed on socket timeout exception: java.net.SocketTimeoutException: 60000
millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
at com.sun.proxy.$Proxy11.openRegion(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.ja
va:533)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1729)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1426)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1398)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1393)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedReg
ionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:48:49,094 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:52,438 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:55,776 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:59,114 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:02,452 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:49:05,797 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one;
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2,
available=1) available servers
2013-12-04 10:49:05,797 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386154145862, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:49:05,797 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:49:05,803 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,804 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:05,804 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
for region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.;
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:05,804 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:46,510 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=0 of 140
failed; retrying after sleep of 1001 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:49:46,514 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:47,517 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 10:50:46,570 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Call openRegion() to ip-
10-232-23-30,60020,1386152980000 has timed out when trying to assign
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a., but the region might already be opened on ip-10-232-23-
30,60020,1386152980000.
java.net.SocketTimeoutException: Call to ip-10-232-23-30/10.232.23.30:60020
failed on socket timeout exception: java.net.SocketTimeoutException: 60000
millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
at com.sun.proxy.$Proxy11.openRegion(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.ja
va:533)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1729)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1426)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1398)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1393)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedReg
ionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267
remote=ip-10-232-23-30/10.232.23.30:60020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:51:46,580 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=1 of 140
failed; retrying after sleep of 1004 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:51:46,583 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:51:47,589 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:52:47,648 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=2 of 140
failed; retrying after sleep of 1002 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34279 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:52:47,651 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:52:48,659 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:53:48,679 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=3 of 140
failed; retrying after sleep of 2011 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34281 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:53:48,681 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:53:50,688 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:50,735 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=4 of 140
failed; retrying after sleep of 2003 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34286 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:54:50,738 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:52,758 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 10:55:52,790 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=5 of 140
failed; retrying after sleep of 4037 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34290 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:55:52,793 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:55:56,812 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:56:56,872 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=6 of 140
failed; retrying after sleep of 4023 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34295 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:56:56,874 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:57:00,883 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:58:00,917 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=7 of 140
failed; retrying after sleep of 8059 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34300 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:58:00,920 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:58:08,944 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:09,003 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=8 of 140
failed; retrying after sleep of 16038 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34304 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:59:09,005 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:25,061 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:00:25,094 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=9 of 140
failed; retrying after sleep of 32233 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34308 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:00:25,097 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:00:57,187 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:01:57,240 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=10 of 140
failed; retrying after sleep of 64260 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34313 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:01:57,242 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:03:01,712 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:04:01,765 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=11 of 140
failed; retrying after sleep of 64281 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34318 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:04:01,768 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:04:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:05:06,257 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:06:06,273 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=12 of 140
failed; retrying after sleep of 64504 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34328 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:06:06,276 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:07:10,650 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:08:10,659 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=13 of 140
failed; retrying after sleep of 64573 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34334 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:08:10,662 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:09:14,773 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:09:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:10:14,832 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=14 of 140
failed; retrying after sleep of 64639 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34342 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:10:14,834 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:11:19,316 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:12:19,350 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=15 of 140
failed; retrying after sleep of 64320 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34350 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:12:19,352 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:13:23,817 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:14:23,876 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=16 of 140
failed; retrying after sleep of 64241 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34358 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:14:23,878 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:14:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 2 region(s) in transition:
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505,
server=null,
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:15:28,187 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:16:28,246 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=17 of 140
failed; retrying after sleep of 64418 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.232.25.198:34367 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:16:28,248 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:30,592 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. state=OFFLINE, ts=1386154045505, server=null
2013-12-04 11:17:30,593 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE
for too long, reassigning
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to a random server
2013-12-04 11:17:30,594 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 11:17:30,601 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:30,601 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:30,601 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one;
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2,
available=2) available servers
2013-12-04 11:17:30,601 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 11:17:30,612 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:30,642 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:32,253 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:37,313 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:40,651 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:43,988 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:47,325 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:50,669 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:54,007 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:57,345 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755,
region=296a4e3f4e890d3f76d822f5c0d6eb0a