You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by zward3x <pa...@gmail.com> on 2010/03/05 18:38:22 UTC
Unable to disable table ...
Hi
we used to run few (2-5) map reduce jobs which imports data (csv files on
hdfs) into one large hbase table (one family only). After few successful
runs we managed to import about 5-10 million records on each run. After few
last runs, we encounter strange behavior of hbase.
First, every task is killed because did not report status in 600 secs by
jobtracker, and we were unable to insert single row into table but there was
no error in any log (jobtracker logs, hdfs logs, hbase logs, ...), also what
is strange that every node (we are having 7 node cluster where each node
have 16MB of memory and 4 cores) did not report any extensive usage of cpu,
memory, HD or net (there was some activity, ~1% cpu, every java instance has
at least as 60% free heap, net was really low, i/o also, ...).
We noticed that job is waiting on table.put, but there was no errors in any
log.
After some investigation i noticed that this table is in disabled state (how
table can go in disable state?)
So, i create another table and start import, everything looks fine.
I also try to enable table from hbase shell, and i got error
"Unable to enable table ..."
Now in master log i'm getting following exception
2010-03-05 18:20:21,056 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 74 on 60000, call regionServerReport(address: 172.17.11.14:60020,
startcode: 1267806532298, load: (requests=0, regions=601, usedHeap=422,
maxHeap=3491), [Lorg.apache.hadoop.hbase.HMsg;@3a0975f2,
[Lorg.apache.hadoop.hbase.HRegionInfo;@431e80a2) from 172.17.11.14:34345:
error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close, pending
open or closing. State:
name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
unassigned=true, pendingOpen=false, open=false, closing=false,
pendingClose=false, closed=false, offlined=false
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to
be closed if it was not already marked as pending close, pending open or
closing. State:
name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
unassigned=true, pendingOpen=false, open=false, closing=false,
pendingClose=false, closed=false, offlined=false
at
org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1650)
at
org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1093)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:659)
at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:454)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
We are using hadoop 0.20.1 and hbase 0.20.3
i found similar problem in this ticket
http://issues.apache.org/jira/browse/HBASE-2064
Any ideas?
Thanks
Zaharije Pasalic
--
View this message in context: http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27796959.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Unable to disable table ...
Posted by zward3x <pa...@gmail.com>.
Just one note:
i scanned .META. table and noticed that there are some OFFLINE regions. Not
sure if this is something related to my problem?
Best
--
View this message in context: http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27797443.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Unable to disable table ...
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Inline.
J-D
On Mon, Mar 8, 2010 at 1:21 PM, zward3x <pa...@gmail.com> wrote:
>
> At the end, i deleted all data and import again and now everything looks
> fine.
>
> But if this can happen in production environment, is there any way to
> recreate this table (in general i can do scan on this table, so i can create
> new table, and copy everything, but this can be long process (we have about
> 40M columns). Is there faster way (can i just copy hdfs files and fake it to
> be in another table)?
Use the bin/add_table.rb tool. It will recreate all the .META. entries
based on what's on HDFS.
>
> Also, is there some kind of state flow when table can goes to disabled
> state?
As I said previously, those regions are probably failed parents of
splits. In details, we use region states eg closed, opened, assigned,
offlined, etc. We use the same state name whether we are disabling a
table or we are doing a region split (the parent region is offlined,
then the daughters are assigned). It's because we mix those that the
table appears to be disabled although it's not. Even if we did use a
different state name, we still need to fix the fact that when an error
occurs on a split that we do not leave the table in an inconsistent
state.
So for the 0.20 branch we have a fix to repair those automatically:
https://issues.apache.org/jira/browse/HBASE-2244
In the 0.21 branch (current svn trunk), we are rewriting the master to
keep a more consistent view of the regions states.
Re: Unable to disable table ...
Posted by zward3x <pa...@gmail.com>.
At the end, i deleted all data and import again and now everything looks
fine.
But if this can happen in production environment, is there any way to
recreate this table (in general i can do scan on this table, so i can create
new table, and copy everything, but this can be long process (we have about
40M columns). Is there faster way (can i just copy hdfs files and fake it to
be in another table)?
Also, is there some kind of state flow when table can goes to disabled
state?
Jean-Daniel Cryans-2 wrote:
>
> Yeah those log lines are just the regions being assigned, nothing of
> value WRT your issue.
>
> J-D
>
> On Fri, Mar 5, 2010 at 1:01 PM, zward3x <pa...@gmail.com>
> wrote:
>>
>> No, restart does not fix.
>>
>> I did not have import logs, here is just portion of master log after
>> recent
>> restart (i can send you an attachment with whole log file (10K lines),
>> but
>> there is no ERROR logs in it):
>>
>> Fri Mar 5 20:29:39 CET 2010 Starting master on hmaster
>> ulimit -n 16384
>> 2010-03-05 20:29:40,495 INFO org.apache.hadoop.hbase.master.HMaster:
>> vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc.,
>> vmVersion=16.0-b13
>
>
--
View this message in context: http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27827781.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Unable to disable table ...
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yeah those log lines are just the regions being assigned, nothing of
value WRT your issue.
J-D
On Fri, Mar 5, 2010 at 1:01 PM, zward3x <pa...@gmail.com> wrote:
>
> No, restart does not fix.
>
> I did not have import logs, here is just portion of master log after recent
> restart (i can send you an attachment with whole log file (10K lines), but
> there is no ERROR logs in it):
>
> Fri Mar 5 20:29:39 CET 2010 Starting master on hmaster
> ulimit -n 16384
> 2010-03-05 20:29:40,495 INFO org.apache.hadoop.hbase.master.HMaster:
> vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc.,
> vmVersion=16.0-b13
Re: Unable to disable table ...
Posted by zward3x <pa...@gmail.com>.
No, restart does not fix.
I did not have import logs, here is just portion of master log after recent
restart (i can send you an attachment with whole log file (10K lines), but
there is no ERROR logs in it):
Fri Mar 5 20:29:39 CET 2010 Starting master on hmaster
ulimit -n 16384
2010-03-05 20:29:40,495 INFO org.apache.hadoop.hbase.master.HMaster:
vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc.,
vmVersion=16.0-b13
2010-03-05 20:29:40,495 INFO org.apache.hadoop.hbase.master.HMaster:
vmInputArguments=[-Xmx3500m, -XX:+HeapDumpOnOutOfMemoryError,
-XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, -Dhbase.log.dir=/home
/hadoop/hbase/logs, -Dhbase.log.file=hbase-hadoop-master-hmaster.log,
-Dhbase.home.dir=/home/hadoop/hbase, -Dhbase.id.str=hadoop,
-Dhbase.root.logger=INFO,DRFA, -Djava.library.path=/home/hadoop/hbase/li
b/native/Linux-amd64-64]
2010-03-05 20:29:40,565 INFO org.apache.hadoop.hbase.master.HMaster: My
address is hmaster:60000
2010-03-05 20:29:40,956 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HMaster, port=60000
2010-03-05 20:29:41,070 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=hmaster
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_18
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/home/hadoop/.jdk1.6.0_18-x64/jre
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hadoop/hbase/conf:/usr/java/default//lib/tools.jar:/home/hadoop/hbase:/home/hadoop/hbase/hbase-0.20.
3.jar:/home/hadoop/hbase/hbase-0.20.3-test.jar:/home/hadoop/hbase/lib/AgileJSON-2009-03-30.jar:/home/hadoop/hbase/lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/hbase/lib/commons-el-from-jetty-5.1.4.jar:
/home/hadoop/hbase/lib/commons-httpclient-3.0.1.jar:/home/hadoop/hbase/lib/commons-logging-1.0.4.jar:/home/hadoop/hbase/lib/commons-logging-api-1.0.4.jar:/home/hadoop/hbase/lib/commons-math-1.1.jar:/hom
e/hadoop/hbase/lib/hadoop-0.20.1-hdfs127-core.jar:/home/hadoop/hbase/lib/hadoop-0.20.1-test.jar:/home/hadoop/hbase/lib/jasper-compiler-5.5.12.jar:/home/hadoop/hbase/lib/jasper-runtime-5.5.12.jar:/home/h
adoop/hbase/lib/jetty-6.1.14.jar:/home/hadoop/hbase/lib/jetty-util-6.1.14.jar:/home/hadoop/hbase/lib/jruby-complete-1.2.0.jar:/home/hadoop/hbase/lib/json.jar:/home/hadoop/hbase/lib/junit-3.8.1.jar:/home
/hadoop/hbase/lib/libthrift-r771587.jar:/home/hadoop/hbase/lib/log4j-1.2.15.jar:/home/hadoop/hbase/lib/lucene-core-3.0.0.jar:/home/hadoop/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hbase/lib/xmle
nc-0.52.jar:/home/hadoop/hbase/lib/zookeeper-3.2.2.jar:/home/hadoop/hbase/lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/hbase/lib/jsp-2.1/jsp-api-2.1.jar:/home/hadoop/hadoop/conf
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hadoop/hbase/lib/native/Linux-amd64-64
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.18-164.el5
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hadoop
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/hadoop
2010-03-05 20:29:41,071 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/home/hadoop/.hbase-0.20.3/logs
2010-03-05 20:29:41,072 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=hmaster:2181 sessionTimeout=180000
watcher=Thread[Thread-1,5,main]
2010-03-05 20:29:41,074 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2010-03-05 20:29:41,082 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server hmaster/172.17.11.1:2181
2010-03-05 20:29:41,083 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected
local=/172.17.11.1:53195 remote=hmaster/172.17.11.1:2181]
2010-03-05 20:29:41,212 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2010-03-05 20:29:41,225 INFO org.apache.hadoop.hbase.master.RegionManager:
-ROOT- region unset (but not set to be reassigned)
2010-03-05 20:29:41,227 INFO org.apache.hadoop.hbase.master.RegionManager:
ROOT inserted into regionsInTransition
2010-03-05 20:29:41,240 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: -8,101 replyHeader:: -8,65949,
0 request:: 0,v{},v{},v{} response:: null
2010-03-05 20:29:41,248 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 1,4 replyHeader:: 1,65949,-101
request:: '/hbase/master,T response::
2010-03-05 20:29:41,280 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 2,1 replyHeader:: 2,65950,-110
request:: '/hbase,,v{s{31,s{'world,'anyone}}},0 response::
2010-03-05 20:29:41,284 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 3,1 replyHeader:: 3,65951,0 r
equest::
'/hbase/master,#3137322e31372e31312e313a3630303030,v{s{31,s{'world,'anyone}}},1
response:: '/hbase/master
2010-03-05 20:29:41,287 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 4,1 replyHeader:: 4,65952,-110
request:: '/hbase,,v{s{31,s{'world,'anyone}}},0 response::
2010-03-05 20:29:41,290 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 5,1 replyHeader:: 5,65953,0 r
equest:: '/hbase/shutdown,#7570,v{s{31,s{'world,'anyone}}},0 response::
'/hbase/shutdown
2010-03-05 20:29:41,301 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 6,3 replyHeader:: 6,65953,0 r
equest:: '/hbase/shutdown,T response::
s{65953,65953,1267817381287,1267817381287,0,0,0,0,2,0,65953}
2010-03-05 20:29:41,304 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 7,4 replyHeader:: 7,65953,0 r
equest:: '/hbase/master,T response::
#3137322e31372e31312e313a3630303030,s{65951,65951,1267817381280,1267817381280,0,0,0,83087679819546624,17,0,65951}
2010-03-05 20:29:41,304 INFO org.apache.hadoop.hbase.master.HMaster: HMaster
initialized on 172.17.11.1:60000
2010-03-05 20:29:41,306 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 8,4 replyHeader:: 8,65953,0 r
equest:: '/hbase/root-region-server,F response::
#3137322e31372e31312e31303a3630303230,s{16,65929,1267025913426,1267815279750,42,0,0,0,18,0,16}
2010-03-05 20:29:41,312 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 9,8 replyHeader:: 9,65953,0 r
equest:: '/hbase/rs,F response:: v{}
2010-03-05 20:29:41,317 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=Master, sessionId=HMaster
2010-03-05 20:29:41,319 INFO
org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2010-03-05 20:29:41,591 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 60010
2010-03-05 20:29:41,592 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 60010
webServer.getConnectors()[0].getLocalPort() returned 60010
2010-03-05 20:29:41,592 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 60010
2010-03-05 20:29:42,460 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder: starting
2010-03-05 20:29:42,461 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60000: starting
2010-03-05 20:29:42,463 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: starting
2010-03-05 20:29:42,463 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: starting
2010-03-05 20:29:42,463 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: starting
2010-03-05 20:29:42,463 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: starting
2010-03-05 20:29:42,464 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: starting
2010-03-05 20:29:42,464 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: starting
2010-03-05 20:29:42,464 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: starting
2010-03-05 20:29:42,464 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: starting
2010-03-05 20:29:42,464 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: starting
2010-03-05 20:29:42,465 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: starting
2010-03-05 20:29:42,465 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60000: starting
2010-03-05 20:29:42,465 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 60000: starting
2010-03-05 20:29:42,465 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60000: starting
2010-03-05 20:29:42,467 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60000: starting
2010-03-05 20:29:42,467 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60000: starting
2010-03-05 20:29:42,467 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 15 on 60000: starting
2010-03-05 20:29:42,468 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60000: starting
2010-03-05 20:29:42,468 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60000: starting
2010-03-05 20:29:42,468 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60000: starting
2010-03-05 20:29:42,468 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60000: starting
2010-03-05 20:29:42,469 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60000: starting
2010-03-05 20:29:42,469 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60000: starting
2010-03-05 20:29:42,469 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60000: starting
2010-03-05 20:29:42,469 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60000: starting
2010-03-05 20:29:42,470 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 25 on 60000: starting
2010-03-05 20:29:42,470 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 26 on 60000: starting
2010-03-05 20:29:42,470 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 27 on 60000: starting
2010-03-05 20:29:42,470 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 28 on 60000: starting
2010-03-05 20:29:42,470 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 29 on 60000: starting
2010-03-05 20:29:42,471 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 30 on 60000: starting
2010-03-05 20:29:42,471 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 31 on 60000: starting
2010-03-05 20:29:42,471 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 32 on 60000: starting
2010-03-05 20:29:42,471 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 33 on 60000: starting
2010-03-05 20:29:42,472 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 35 on 60000: starting
2010-03-05 20:29:42,472 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 36 on 60000: starting
2010-03-05 20:29:42,472 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 37 on 60000: starting
2010-03-05 20:29:42,472 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 38 on 60000: starting
2010-03-05 20:29:42,473 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 39 on 60000: starting
2010-03-05 20:29:42,473 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 40 on 60000: starting
2010-03-05 20:29:42,473 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 41 on 60000: starting
2010-03-05 20:29:42,473 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 42 on 60000: starting
2010-03-05 20:29:42,473 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 43 on 60000: starting
2010-03-05 20:29:42,474 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 44 on 60000: starting
2010-03-05 20:29:42,474 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 45 on 60000: starting
2010-03-05 20:29:42,474 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60000: starting
2010-03-05 20:29:42,474 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 46 on 60000: starting
2010-03-05 20:29:42,475 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 47 on 60000: starting
2010-03-05 20:29:42,475 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 34 on 60000: starting
2010-03-05 20:29:42,476 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 48 on 60000: starting
2010-03-05 20:29:42,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 49 on 60000: starting
2010-03-05 20:29:42,484 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 51 on 60000: starting
2010-03-05 20:29:42,484 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 52 on 60000: starting
2010-03-05 20:29:42,484 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 53 on 60000: starting
2010-03-05 20:29:42,484 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 54 on 60000: starting
2010-03-05 20:29:42,485 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 55 on 60000: starting
2010-03-05 20:29:42,485 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 56 on 60000: starting
2010-03-05 20:29:42,485 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 57 on 60000: starting
2010-03-05 20:29:42,485 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 58 on 60000: starting
2010-03-05 20:29:42,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 59 on 60000: starting
2010-03-05 20:29:42,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 60 on 60000: starting
2010-03-05 20:29:42,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 61 on 60000: starting
2010-03-05 20:29:42,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 63 on 60000: starting
2010-03-05 20:29:42,487 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 64 on 60000: starting
2010-03-05 20:29:42,487 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 65 on 60000: starting
2010-03-05 20:29:42,487 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 62 on 60000: starting
2010-03-05 20:29:42,494 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 99 on 60000: starting
2010-03-05 20:29:42,494 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 98 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 97 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 96 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 95 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 94 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 93 on 60000: starting
2010-03-05 20:29:42,493 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 92 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 90 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 91 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 89 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 50 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 88 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 87 on 60000: starting
2010-03-05 20:29:42,492 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 86 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 85 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 84 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 83 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 82 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 81 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 80 on 60000: starting
2010-03-05 20:29:42,491 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 79 on 60000: starting
2010-03-05 20:29:42,490 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 78 on 60000: starting
2010-03-05 20:29:42,490 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 77 on 60000: starting
2010-03-05 20:29:42,490 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 76 on 60000: starting
2010-03-05 20:29:42,490 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 75 on 60000: starting
2010-03-05 20:29:42,490 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 74 on 60000: starting
2010-03-05 20:29:42,489 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 73 on 60000: starting
2010-03-05 20:29:42,489 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 72 on 60000: starting
2010-03-05 20:29:42,489 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 71 on 60000: starting
2010-03-05 20:29:42,489 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 70 on 60000: starting
2010-03-05 20:29:42,488 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 69 on 60000: starting
2010-03-05 20:29:42,488 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 68 on 60000: starting
2010-03-05 20:29:42,488 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 67 on 60000: starting
2010-03-05 20:29:42,487 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 66 on 60000: starting
2010-03-05 20:29:42,524 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h13,60020,1267817459503
2010-03-05 20:29:42,525 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h00,60020,1267817431514
2010-03-05 20:29:42,527 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h04,60020,1267817430492
2010-03-05 20:29:42,527 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h02,60020,1267817415531
2010-03-05 20:29:42,532 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 10,5 replyHeader:: 10,65969,0
request:: '/hbase/rs/1267817430492,#3137322e31372e31312e31343a3630303230,-1
response::
s{65966,65969,1267817382518,1267817382528,1,0,0,83087679819546628,18,0,65966}
2010-03-05 20:29:42,534 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 11,5 replyHeader:: 11,65970,0
request:: '/hbase/rs/1267817459503,#3137322e31372e31312e32333a3630303230,-1
response::
s{65965,65970,1267817382516,1267817382529,1,0,0,83087679819546625,18,0,65965}
2010-03-05 20:29:42,535 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 12,5 replyHeader:: 12,65971,0
request:: '/hbase/rs/1267817415531,#3137322e31372e31312e31323a3630303230,-1
response::
s{65968,65971,1267817382521,1267817382529,1,0,0,83087679819546626,18,0,65968}
2010-03-05 20:29:42,537 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 13,5 replyHeader:: 13,65972,0
request:: '/hbase/rs/1267817431514,#3137322e31372e31312e31303a3630303230,-1
response::
s{65967,65972,1267817382518,1267817382530,1,0,0,83087679819546627,18,0,65967}
2010-03-05 20:29:42,538 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 14,4 replyHeader:: 14,65972,0
request:: '/hbase/rs/1267817430492,T response::
#3137322e31372e31312e31343a3630303230,s{65966,65969,1267817382518,1267817382528,1,0,0,83087679819546628,18,0,65966}
2010-03-05 20:29:42,539 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 15,4 replyHeader:: 15,65972,0
request:: '/hbase/rs/1267817459503,T response::
#3137322e31372e31312e32333a3630303230,s{65965,65970,1267817382516,1267817382529,1,0,0,83087679819546625,18,0,65965}
2010-03-05 20:29:42,541 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 16,4 replyHeader:: 16,65972,0
request:: '/hbase/rs/1267817415531,T response::
#3137322e31372e31312e31323a3630303230,s{65968,65971,1267817382521,1267817382529,1,0,0,83087679819546626,18,0,65968}
2010-03-05 20:29:42,542 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 17,4 replyHeader:: 17,65972,0
request:: '/hbase/rs/1267817431514,T response::
#3137322e31372e31312e31303a3630303230,s{65967,65972,1267817382518,1267817382530,1,0,0,83087679819546627,18,0,65967}
2010-03-05 20:29:42,588 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h17,60020,1267817458545
2010-03-05 20:29:42,591 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 18,5 replyHeader:: 18,65975,0
request:: '/hbase/rs/1267817458545,#3137322e31372e31312e32373a3630303230,-1
response::
s{65974,65975,1267817382585,1267817382589,1,0,0,83087679819546629,18,0,65974}
2010-03-05 20:29:42,593 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 19,4 replyHeader:: 19,65975,0
request:: '/hbase/rs/1267817458545,T response::
#3137322e31372e31312e32373a3630303230,s{65974,65975,1267817382585,1267817382589,1,0,0,83087679819546629,18,0,65974}
2010-03-05 20:29:42,658 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h15,60020,1267817464640
2010-03-05 20:29:42,661 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: h14,60020,1267817458623
2010-03-05 20:29:42,661 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 20,5 replyHeader:: 20,65980,0
request:: '/hbase/rs/1267817464640,#3137322e31372e31312e32353a3630303230,-1
response::
s{65978,65980,1267817382654,1267817382659,1,0,0,83087679819546630,18,0,65978}
2010-03-05 20:29:42,666 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 21,5 replyHeader:: 21,65981,0
request:: '/hbase/rs/1267817458623,#3137322e31372e31312e32343a3630303230,-1
response::
s{65979,65981,1267817382657,1267817382663,1,0,0,83087679819546631,18,0,65979}
2010-03-05 20:29:42,667 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 22,4 replyHeader:: 22,65981,0
request:: '/hbase/rs/1267817464640,T response::
#3137322e31372e31312e32353a3630303230,s{65978,65980,1267817382654,1267817382659,1,0,0,83087679819546630,18,0,65978}
2010-03-05 20:29:42,670 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 23,4 replyHeader:: 23,65981,0
request:: '/hbase/rs/1267817458623,T response::
#3137322e31372e31312e32343a3630303230,s{65979,65981,1267817382657,1267817382663,1,0,0,83087679819546631,18,0,65979}
2010-03-05 20:29:43,231 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region -ROOT-,,0 to h13,60020,1267817459503
2010-03-05 20:29:43,471 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:43,512 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:43,520 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:43,528 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:43,543 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:43,806 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2010-03-05 20:29:44,239 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN: -ROOT-,,0 from h13,60020,1267817459503; 1 of 1
2010-03-05 20:29:44,242 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 24,1 replyHeader:: 24,65983,-1
10 request:: '/hbase,,v{s{31,s{'world,'anyone}}},0 response::
2010-03-05 20:29:44,244 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 25,1 replyHeader:: 25,65984,-1
10 request:: '/hbase/safe-mode,,v{s{31,s{'world,'anyone}}},0 response::
2010-03-05 20:29:44,244 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of
safe mode in ZooKeeper:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
= NodeExists for /hbase/safe-mode
2010-03-05 20:29:44,245 INFO
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move
on
2010-03-05 20:29:44,245 INFO org.apache.hadoop.hbase.master.RegionManager:
exiting safe mode
2010-03-05 20:29:44,247 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 26,1 replyHeader:: 26,65985,-1
10 request:: '/hbase,,v{s{31,s{'world,'anyone}}},0 response::
2010-03-05 20:29:44,249 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 27,3 replyHeader:: 27,65985,0
request:: '/hbase/root-region-server,F response::
s{16,65929,1267025913426,1267815279750,42,0,0,0,18,0,16}
2010-03-05 20:29:44,251 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply
sessionid:0x1272fcdf8db0000, packet:: clientPath:null serverPath:null
finished:false header:: 28,5 replyHeader:: 28,65986,0
request::
'/hbase/root-region-server,#3137322e31372e31312e32333a3630303230,-1
response:: s{16,65986,1267025913426,1267817384249,43,0,0,0,18,0,16}
2010-03-05 20:29:44,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 172.17.11.23:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-03-05 20:29:44,315 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
172.17.11.23:60020, regionname: -ROOT-,,0, startKey: <>} compl
ete
2010-03-05 20:29:44,477 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region .META.,,1 to h00,60020,1267817431514
2010-03-05 20:29:45,485 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN: .META.,,1 from h00,60020,1267817431514;
1 of 2
2010-03-05 20:29:45,485 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN: .META.,,1 from h00,60020,1267817431514; 2 of 2
2010-03-05 20:29:45,488 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: .META.,,1 open on
172.17.11.10:60020
2010-03-05 20:29:45,490 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row .META.,,1
in region -ROOT-,,0 with startcode=1267817431514, server=172.17.11.10:60020
2010-03-05 20:30:41,231 INFO org.apache.hadoop.hbase.master.ServerManager: 7
region servers, 0 dead, average load 0.2857142857142857
2010-03-05 20:30:41,329 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 172.17.11.23:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-03-05 20:30:41,334 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 172.17.11.10:60020,
regionname: .META.,,1, startKey: <>}
2010-03-05 20:30:41,342 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
172.17.11.23:60020, regionname: -ROOT-,,0, startKey: <>} compl
ete
2010-03-05 20:30:41,493 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,003e8217-7a4a-4b9f-af40-391fb27e037e,1267765880682
to h13,60020,1267817459503
2010-03-05 20:30:41,729 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0174b020-2a9f-41a6-8b3e-b769f8a2f63b,1267764837953
to h00,60020,1267817431514
2010-03-05 20:30:41,729 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,02e99c1d-fe93-4018-bd2a-f68d465c4dd2,1267763953052
to h00,60020,1267817431514
2010-03-05 20:30:41,729 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,00bc4fab-f899-4c7c-90e6-125acf33a801,1267764465900
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,001fd89f-a3a5-47be-87f1-6455c78375a1,1267760425816
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,009d0e9e-7fc0-4526-b3bc-2ae4131ad067,1267763241820
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,02ab7f4d-a729-4a77-a3dd-efbbd183f1ea,1267764236375
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,01563438-a486-4919-b8c8-d2068eedd456,1267769710466
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0194494b-eb17-48ee-8853-26abad6e4232,1267764837953
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,007d8ba3-7d3d-48a0-a068-f8d876235d72,1267763241820
to h00,60020,1267817431514
2010-03-05 20:30:41,730 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,028cfb00-0b0d-4b60-afda-75b8e3d2de67,1267766056457
to h00,60020,1267817431514
2010-03-05 20:30:41,738 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,03088739-144a-40da-8054-b2250e7bcefa,1267763953052
to h02,60020,1267817415531
2010-03-05 20:30:41,738 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,00db6a30-8b27-428f-a36c-7d619f2de614,1267764465900
to h02,60020,1267817415531
2010-03-05 20:30:41,738 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0137bdcf-40d7-4e5b-9068-37cfcf4a38d3,1267769710466
to h02,60020,1267817415531
2010-03-05 20:30:41,742 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,01d2b5a7-6312-4b98-b7fb-79a56ead0437,1267763498464
to h15,60020,1267817464640
2010-03-05 20:30:41,742 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,026e4aee-4fe9-4cae-a538-93c44285940e,1267766056457
to h15,60020,1267817464640
2010-03-05 20:30:41,748 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0327ef47-6db8-4998-b14f-ce31c5f168e4,1267765070252
to h04,60020,1267817430492
2010-03-05 20:30:41,748 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,00fac00d-415f-4366-bdf7-0731297b0575,1267766316982
to h04,60020,1267817430492
2010-03-05 20:30:41,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0210f5d0-2076-4b5a-9f44-5a66ec1718ce,1267762893844
to h17,60020,1267817458545
2010-03-05 20:30:41,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,01b36f82-3a62-47e9-9a78-0ce9379c9454,1267763498464
to h17,60020,1267817458545
2010-03-05 20:30:42,031 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,057b1557-6de1-4b76-b06b-c33e9f02a181,1267766543053
to h14,60020,1267817458623
2010-03-05 20:30:42,031 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,051c4f8e-ef67-4da1-83d6-f39ff85cae55,1267764379969
to h14,60020,1267817458623
2010-03-05 20:30:42,031 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,03c46053-612c-4945-95cb-655234ec7b7b,1267766307380
to h14,60020,1267817458623
2010-03-05 20:30:42,031 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,03a4d5c0-bda8-4295-a83a-2f5eafe1667f,1267766307380
to h14,60020,1267817458623
2010-03-05 20:30:42,031 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,045ef692-32c5-4971-ad15-59bc9beac6c5,1267764516986
to h14,60020,1267817458623
2010-03-05 20:30:42,499 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,003e8217-7a4a-4b9f-af40-391fb27e037e,1267765880682 from
h13,60020,1267817459503; 1
of 2
2010-03-05 20:30:42,499 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,003e8217-7a4a-4b9f-af40-391fb27e037e,1267765880682 from
h13,60020,1267817459503; 2 of 2
2010-03-05 20:30:42,500 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,003e8217-7a4a-4b9f-af40-391fb27e037e,1267765880682 open on
172.17.11.23:60020
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,06d395c7-0466-477e-9817-a9aeaac931fc,1267763951238
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,086aa086-27f7-4cf1-9cf3-ce70cb248874,1267770284344
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,095f11ce-61b6-4a3a-9c40-d51caa70d332,1267767105399
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0402a4df-1c61-474a-b8d0-39dd718e2044,1267759340050
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,055be6c1-8a29-4b03-a969-8642502a3d39,1267766543053
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,094076e7-e38c-49b4-9903-0db51e7e98b8,1267767105399
to h13,60020,1267817459503
2010-03-05 20:30:42,503 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0385ac16-a585-47ef-8602-27dafc735171,1267760440389
to h13,60020,1267817459503
2010-03-05 20:30:42,515 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,003e8217-7a4a-4b9f-af40-391fb27e037e,1267765880682 in region
.META.,,1 with startcode=126781745950
3, server=172.17.11.23:60020
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,0174b020-2a9f-41a6-8b3e-b769f8a2f63b,1267764837953 from
h00,60020,1267817431514; 1
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,02e99c1d-fe93-4018-bd2a-f68d465c4dd2,1267763953052 from
h00,60020,1267817431514; 2
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,00bc4fab-f899-4c7c-90e6-125acf33a801,1267764465900 from
h00,60020,1267817431514; 3
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,001fd89f-a3a5-47be-87f1-6455c78375a1,1267760425816 from
h00,60020,1267817431514; 4
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,009d0e9e-7fc0-4526-b3bc-2ae4131ad067,1267763241820 from
h00,60020,1267817431514; 5
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,02ab7f4d-a729-4a77-a3dd-efbbd183f1ea,1267764236375 from
h00,60020,1267817431514; 6
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,01563438-a486-4919-b8c8-d2068eedd456,1267769710466 from
h00,60020,1267817431514; 7
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,0194494b-eb17-48ee-8853-26abad6e4232,1267764837953 from
h00,60020,1267817431514; 8
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,007d8ba3-7d3d-48a0-a068-f8d876235d72,1267763241820 from
h00,60020,1267817431514; 9
of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,028cfb00-0b0d-4b60-afda-75b8e3d2de67,1267766056457 from
h00,60020,1267817431514; 1
0 of 18
2010-03-05 20:30:42,746 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,0174b020-2a9f-41a6-8b3e-b769f8a2f63b,1267764837953 from
h00,60020,1267817431514; 11 of 18
2010-03-05 20:30:42,746 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,0174b020-2a9f-41a6-8b3e-b769f8a2f63b,1267764837953 open on
172.17.11.10:60020
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,02e99c1d-fe93-4018-bd2a-f68d465c4dd2,1267763953052 from
h00,60020,1267817431514; 12 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,00bc4fab-f899-4c7c-90e6-125acf33a801,1267764465900 from
h00,60020,1267817431514; 13 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,001fd89f-a3a5-47be-87f1-6455c78375a1,1267760425816 from
h00,60020,1267817431514; 14 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,009d0e9e-7fc0-4526-b3bc-2ae4131ad067,1267763241820 from
h00,60020,1267817431514; 15 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,02ab7f4d-a729-4a77-a3dd-efbbd183f1ea,1267764236375 from
h00,60020,1267817431514; 16 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,01563438-a486-4919-b8c8-d2068eedd456,1267769710466 from
h00,60020,1267817431514; 17 of 18
2010-03-05 20:30:42,747 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,0194494b-eb17-48ee-8853-26abad6e4232,1267764837953 from
h00,60020,1267817431514; 18 of 18
2010-03-05 20:30:42,749 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,00db6a30-8b27-428f-a36c-7d619f2de614,1267764465900 from
h02,60020,1267817415531; 1
of 5
2010-03-05 20:30:42,749 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,0137bdcf-40d7-4e5b-9068-37cfcf4a38d3,1267769710466 from
h02,60020,1267817415531; 2
of 5
2010-03-05 20:30:42,749 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,03088739-144a-40da-8054-b2250e7bcefa,1267763953052 from
h02,60020,1267817415531; 3 of 5
2010-03-05 20:30:42,749 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,00db6a30-8b27-428f-a36c-7d619f2de614,1267764465900 from
h02,60020,1267817415531; 4 of 5
2010-03-05 20:30:42,750 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,0137bdcf-40d7-4e5b-9068-37cfcf4a38d3,1267769710466 from
h02,60020,1267817415531; 5 of 5
2010-03-05 20:30:42,751 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,0174b020-2a9f-41a6-8b3e-b769f8a2f63b,1267764837953 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,751 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0a96f336-c073-4004-bebe-928784466ab4,1267766546021
to h02,60020,1267817415531
2010-03-05 20:30:42,751 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0b8e907b-ea04-488e-8dc9-2dc67d8a9b18,1267767299904
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0b5118dd-9ecf-467f-82f1-8d5698776ee5,1267763523333
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0ca2ecaf-7ed0-4def-941f-05d756422675,1267765048321
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0cff0386-0fc3-4ba0-9b35-d01b63fdac5a,1267765074750
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0712f2fc-6af6-4211-a0aa-a64137ec7021,1267757335048
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,01d2b5a7-6312-4b98-b7fb-79a56ead0437,1267763498464 from
h15,60020,1267817464640; 1
of 4
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,026e4aee-4fe9-4cae-a538-93c44285940e,1267766056457 from
h15,60020,1267817464640; 2
of 4
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,01d2b5a7-6312-4b98-b7fb-79a56ead0437,1267763498464 from
h15,60020,1267817464640; 3 of 4
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,005e5016-ed26-4f59-9fa7-09da4b90f6c9,1267765880682
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0c655529-9eba-4bc1-9cee-4dc14e016496,1267768452376
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,04fc5a4d-8c41-4c8d-9913-776cf332ebc9,1267759261500
to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,,1267760425816 to h02,60020,1267817415531
2010-03-05 20:30:42,752 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,02e99c1d-fe93-4018-bd2a-f68d465c4dd2,1267763953052 open on
172.17.11.10:60020
2010-03-05 20:30:42,753 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,026e4aee-4fe9-4cae-a538-93c44285940e,1267766056457 from
h15,60020,1267817464640; 4 of 4
2010-03-05 20:30:42,754 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0b126b17-1db3-4c4d-8d04-b5435927415d,1267762899333
to h15,60020,1267817464640
2010-03-05 20:30:42,754 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,023040aa-410a-469c-9be9-1061a62cff0e,1267766313495
to h15,60020,1267817464640
2010-03-05 20:30:42,754 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0c282ebc-0c1c-46b3-83e8-0294cf0bd0b2,1267771283173
to h15,60020,1267817464640
2010-03-05 20:30:42,754 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,099cf5d8-e361-48ff-b026-c5a095f4be66,1267764817345
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,06765298-acc0-462c-9722-fdc7f050495c,1267767710223
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,06b41bec-bd3d-4306-895d-a80302a5bec8,1267760563117
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0617e6da-7b76-440c-827f-be03da47ac1d,1267756530359
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,03e3dfc6-d587-4a0f-9870-735d94382606,1267759340050
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0a1a37c9-44e8-4504-8183-040d2da1bfd0,1267769981731
to h15,60020,1267817464640
2010-03-05 20:30:42,755 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,07724586-784b-4f71-a92f-1f94758d5a96,1267757337435
to h15,60020,1267817464640
2010-03-05 20:30:42,758 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,00fac00d-415f-4366-bdf7-0731297b0575,1267766316982 from
h04,60020,1267817430492; 1
of 3
2010-03-05 20:30:42,758 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,0327ef47-6db8-4998-b14f-ce31c5f168e4,1267765070252 from
h04,60020,1267817430492; 2 of 3
2010-03-05 20:30:42,759 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,00fac00d-415f-4366-bdf7-0731297b0575,1267766316982 from
h04,60020,1267817430492; 3 of 3
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0d3d0e25-ecd6-4a2b-a513-9f0107ece3e7,1267768732514
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,088931a5-7c26-4c78-9af2-1df3ef317fce,1267766473205
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0347330f-5a47-4865-b651-f45593a0532e,1267765070252
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,047e2d63-5233-4211-b896-09ae7ea351e8,1267764516986
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,02e99c1d-fe93-4018-bd2a-f68d465c4dd2,1267763953052 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,04bd2cce-b8d6-4c5b-9a77-ff4d52ae1467,1267765025421
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0a58c9c2-69b2-4240-88a2-83fc29045f40,1267766480259
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0365e2bd-2001-48c2-adcc-0f3817d43635,1267760440389
to h04,60020,1267817430492
2010-03-05 20:30:42,760 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0c46b891-a4c0-42e0-b95a-e5dce9be8a1b,1267771283173
to h04,60020,1267817430492
2010-03-05 20:30:42,761 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0bcbad74-3a51-467a-bf7a-2b035be01433,1267766309981
to h04,60020,1267817430492
2010-03-05 20:30:42,761 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,01192a39-a4a7-4548-a601-de9180e3a9db,1267766316982
to h04,60020,1267817430492
2010-03-05 20:30:42,761 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,00bc4fab-f899-4c7c-90e6-125acf33a801,1267764465900 open on
172.17.11.10:60020
2010-03-05 20:30:42,762 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,00bc4fab-f899-4c7c-90e6-125acf33a801,1267764465900 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,763 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,009d0e9e-7fc0-4526-b3bc-2ae4131ad067,1267763241820 open on
172.17.11.10:60020
2010-03-05 20:30:42,765 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_PROCESS_OPEN:
profiles,01b36f82-3a62-47e9-9a78-0ce9379c9454,1267763498464 from
h17,60020,1267817458545; 1
of 3
2010-03-05 20:30:42,765 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,0210f5d0-2076-4b5a-9f44-5a66ec1718ce,1267762893844 from
h17,60020,1267817458545; 2 of 3
2010-03-05 20:30:42,765 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN:
profiles,01b36f82-3a62-47e9-9a78-0ce9379c9454,1267763498464 from
h17,60020,1267817458545; 3 of 3
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0ce03461-3ad7-4f21-abbe-020ffad79704,1267765074750
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,084c2b66-64eb-49d5-a6e9-40b7b7903f38,1267770284344
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,08c5f98f-438d-4c46-bfdd-5d856d988f6c,1267766314848
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,07f04087-baa4-49b6-9d8f-d5ceecedfa0e,1267766366416
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0657a2f7-07fe-4e27-92c1-a99e5ffbb078,1267767710223
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,080ef6ba-322d-43a5-9f52-d79ddf3ca4e5,1267766065346
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,04402f70-0874-4516-99f7-d9d15890cca9,1267767531433
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,049dfc2c-d9d0-4011-82f6-8b1ac350ba04,1267765025421
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,09db33db-cbd5-4e23-bb94-c74ab0e88c41,1267757975396
to h17,60020,1267817458545
2010-03-05 20:30:42,766 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region profiles,0ab57d4b-ae7d-4830-9522-981d97a1778a,1267767040114
to h17,60020,1267817458545
2010-03-05 20:30:42,773 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,009d0e9e-7fc0-4526-b3bc-2ae4131ad067,1267763241820 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,773 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,02ab7f4d-a729-4a77-a3dd-efbbd183f1ea,1267764236375 open on
172.17.11.10:60020
2010-03-05 20:30:42,775 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,02ab7f4d-a729-4a77-a3dd-efbbd183f1ea,1267764236375 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,775 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,001fd89f-a3a5-47be-87f1-6455c78375a1,1267760425816 open on
172.17.11.10:60020
2010-03-05 20:30:42,789 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,001fd89f-a3a5-47be-87f1-6455c78375a1,1267760425816 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,789 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,01563438-a486-4919-b8c8-d2068eedd456,1267769710466 open on
172.17.11.10:60020
2010-03-05 20:30:42,792 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,01563438-a486-4919-b8c8-d2068eedd456,1267769710466 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,793 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,0194494b-eb17-48ee-8853-26abad6e4232,1267764837953 open on
172.17.11.10:60020
2010-03-05 20:30:42,802 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,0194494b-eb17-48ee-8853-26abad6e4232,1267764837953 in region
.META.,,1 with startcode=126781743151
4, server=172.17.11.10:60020
2010-03-05 20:30:42,802 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,03088739-144a-40da-8054-b2250e7bcefa,1267763953052 open on
172.17.11.12:60020
2010-03-05 20:30:42,804 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
profiles,03088739-144a-40da-8054-b2250e7bcefa,1267763953052 in region
.META.,,1 with startcode=126781741553
1, server=172.17.11.12:60020
2010-03-05 20:30:42,804 INFO
org.apache.hadoop.hbase.master.RegionServerOperation:
profiles,00db6a30-8b27-428f-a36c-7d619f2de614,1267764465900 open on
172.17.11.12:60020
2010-03-05 20:30:42,817 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: Updated row
Jean-Daniel Cryans-2 wrote:
>
> Zaharije,
>
> Looks like your table somehow got into an inconsistant state...
> although the offline regions are probably just split parents.
>
> Does a full HBase restart fixes the problem?
>
> No, it does not.
>
> Can we see your full master log from during the import and just after it?
>
>
>
> Thx,
>
> J-D
>
> On Fri, Mar 5, 2010 at 9:38 AM, zward3x <pa...@gmail.com>
> wrote:
>>
>> Hi
>>
>> we used to run few (2-5) map reduce jobs which imports data (csv files on
>> hdfs) into one large hbase table (one family only). After few successful
>> runs we managed to import about 5-10 million records on each run. After
>> few
>> last runs, we encounter strange behavior of hbase.
>>
>> First, every task is killed because did not report status in 600 secs by
>> jobtracker, and we were unable to insert single row into table but there
>> was
>> no error in any log (jobtracker logs, hdfs logs, hbase logs, ...), also
>> what
>> is strange that every node (we are having 7 node cluster where each node
>> have 16MB of memory and 4 cores) did not report any extensive usage of
>> cpu,
>> memory, HD or net (there was some activity, ~1% cpu, every java instance
>> has
>> at least as 60% free heap, net was really low, i/o also, ...).
>>
>> We noticed that job is waiting on table.put, but there was no errors in
>> any
>> log.
>>
>> After some investigation i noticed that this table is in disabled state
>> (how
>> table can go in disable state?)
>>
>> So, i create another table and start import, everything looks fine.
>>
>> I also try to enable table from hbase shell, and i got error
>>
>> "Unable to enable table ..."
>>
>> Now in master log i'm getting following exception
>>
>> 2010-03-05 18:20:21,056 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server
>> handler 74 on 60000, call regionServerReport(address: 172.17.11.14:60020,
>> startcode: 1267806532298, load: (requests=0, regions=601, usedHeap=422,
>> maxHeap=3491), [Lorg.apache.hadoop.hbase.HMsg;@3a0975f2,
>> [Lorg.apache.hadoop.hbase.HRegionInfo;@431e80a2) from 172.17.11.14:34345:
>> error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
>> region to be closed if it was not already marked as pending close,
>> pending
>> open or closing. State:
>> name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
>> unassigned=true, pendingOpen=false, open=false, closing=false,
>> pendingClose=false, closed=false, offlined=false
>> java.io.IOException: java.lang.IllegalStateException: Cannot set a region
>> to
>> be closed if it was not already marked as pending close, pending open or
>> closing. State:
>> name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
>> unassigned=true, pendingOpen=false, open=false, closing=false,
>> pendingClose=false, closed=false, offlined=false
>> at
>> org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1650)
>> at
>> org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1093)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:659)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:454)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
>> at
>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>>
>>
>> We are using hadoop 0.20.1 and hbase 0.20.3
>>
>> i found similar problem in this ticket
>> http://issues.apache.org/jira/browse/HBASE-2064
>>
>> Any ideas?
>>
>> Thanks
>> Zaharije Pasalic
>> --
>> View this message in context:
>> http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27796959.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27799139.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Unable to disable table ...
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Zaharije,
Looks like your table somehow got into an inconsistant state...
although the offline regions are probably just split parents.
Does a full HBase restart fixes the problem?
Can we see your full master log from during the import and just after it?
Thx,
J-D
On Fri, Mar 5, 2010 at 9:38 AM, zward3x <pa...@gmail.com> wrote:
>
> Hi
>
> we used to run few (2-5) map reduce jobs which imports data (csv files on
> hdfs) into one large hbase table (one family only). After few successful
> runs we managed to import about 5-10 million records on each run. After few
> last runs, we encounter strange behavior of hbase.
>
> First, every task is killed because did not report status in 600 secs by
> jobtracker, and we were unable to insert single row into table but there was
> no error in any log (jobtracker logs, hdfs logs, hbase logs, ...), also what
> is strange that every node (we are having 7 node cluster where each node
> have 16MB of memory and 4 cores) did not report any extensive usage of cpu,
> memory, HD or net (there was some activity, ~1% cpu, every java instance has
> at least as 60% free heap, net was really low, i/o also, ...).
>
> We noticed that job is waiting on table.put, but there was no errors in any
> log.
>
> After some investigation i noticed that this table is in disabled state (how
> table can go in disable state?)
>
> So, i create another table and start import, everything looks fine.
>
> I also try to enable table from hbase shell, and i got error
>
> "Unable to enable table ..."
>
> Now in master log i'm getting following exception
>
> 2010-03-05 18:20:21,056 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 74 on 60000, call regionServerReport(address: 172.17.11.14:60020,
> startcode: 1267806532298, load: (requests=0, regions=601, usedHeap=422,
> maxHeap=3491), [Lorg.apache.hadoop.hbase.HMsg;@3a0975f2,
> [Lorg.apache.hadoop.hbase.HRegionInfo;@431e80a2) from 172.17.11.14:34345:
> error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
> region to be closed if it was not already marked as pending close, pending
> open or closing. State:
> name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
> unassigned=true, pendingOpen=false, open=false, closing=false,
> pendingClose=false, closed=false, offlined=false
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to
> be closed if it was not already marked as pending close, pending open or
> closing. State:
> name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
> unassigned=true, pendingOpen=false, open=false, closing=false,
> pendingClose=false, closed=false, offlined=false
> at
> org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1650)
> at
> org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1093)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:659)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:454)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
>
> We are using hadoop 0.20.1 and hbase 0.20.3
>
> i found similar problem in this ticket
> http://issues.apache.org/jira/browse/HBASE-2064
>
> Any ideas?
>
> Thanks
> Zaharije Pasalic
> --
> View this message in context: http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27796959.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>