You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2014/01/09 04:23:56 UTC
Re: graceful_stop.sh hung
Can you check region server log on inspur253.deu.edu.cn,60020,1388053123213
?
Cheers
On Wed, Jan 8, 2014 at 5:34 PM, hzwangxx <wh...@163.com> wrote:
> Hi, Ted
> This is the master log:
>
> *2014-01-08 18:40:48,640 [IPC Server handler 37 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Addeu.edu.cnded move plan
> hri=test,|a1bd417af20749*
> *110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751.78c953d53f6498664d9a067701a7e7d7.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789,
> d*
> *est=inspur255.deu.edu.cn
> <http://inspur255.deu.edu.cn>,60020,1388056934052, running balancer*
> *2014-01-08 18:40:48,860 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|a1bd4*
> *17af20749110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751.78c953d53f6498664d9a067701a7e7d7.
> that was online on inspur255.deu.edu.cn <http://inspur255.deu.edu.cn>*
> *,60020,1388056934052*
> *2014-01-08 18:40:50,690 [IPC Server handler 44 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|a786b578f29c43*
> *1374b62ca7df559277|1380621867798|7273081556605895,1382673452778.c621b3bf29262ca5248c03a8d6ebb41e.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789,
> d*
> *est=inspur253.deu.edu.cn
> <http://inspur253.deu.edu.cn>,60020,1388053123213, running balancer*
> *2014-01-08 18:40:51,078 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|a786b*
> *578f29c431374b62ca7df559277|1380621867798|7273081556605895,1382673452778.c621b3bf29262ca5248c03a8d6ebb41e.
> that was online on inspur253.deu.edu.cn <http://inspur253.deu.edu.cn>*
> *,60020,1388053123213*
> *2014-01-08 18:40:53,944 [IPC Server handler 46 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|bucket-ynote-o*
> *nline|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098946420,1386199418157.4dad873a6af4d3a9809339281c3cb34c.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60*
> *020,1388050383789, dest=inspur254.deu.edu.cn
> <http://inspur254.deu.edu.cn>,60020,1388054917364, running balancer*
> *2014-01-08 18:40:55,416 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|bucke*
> *t-ynote-online|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098946420,1386199418157.4dad873a6af4d3a9809339281c3cb34c.
> that was online on ins*
> *pur254.deu.edu.cn <http://pur254.deu.edu.cn>,60020,1388054917364*
> *2014-01-08 18:40:57,067 [IPC Server handler 7 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|c4617a74baabdd0*
> *6f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237.0e311941f5ff202bcefe57aa4079a188.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789,
> de*
> *st=inspur253.deu.edu.cn
> <http://inspur253.deu.edu.cn>,60020,1388053123213, running balancer*
> *2014-01-08 18:40:57,511 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|c4617*
> *a74baabdd06f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237.0e311941f5ff202bcefe57aa4079a188.
> that was online on inspur253.deu.edu.cn <http://inspur253.deu.edu.cn>*
> *,60020,1388053123213*
> *2014-01-08 18:41:00,143 [IPC Server handler 23 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|coursera-video*
> *|0dc2a6efeba02749d6187481d8f18357|1380039238151|41650362285832984,1380853043651.3a955559fb65caf32a05e18b8b6b93f8.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,*
> *1388050383789, dest=inspur308.deu.edu.cn
> <http://inspur308.deu.edu.cn>,60020,1388059770705, running balancer*
> *2014-01-08 18:41:00,989 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|cours*
> *era-video|0dc2a6efeba02749d6187481d8f18357|1380039238151|41650362285832984,1380853043651.3a955559fb65caf32a05e18b8b6b93f8.
> that was online on inspur3*
> *08.deu.edu.cn <http://08.deu.edu.cn>,60020,1388059770705*
> *2014-01-08 18:41:01,904 [935521285@qtp-711761606-0] WARN
> org.apache.hadoop.conf.Configuration - fs.default.name
> <http://fs.default.name> is deprecated. Instead, use fs.defau*
> *ltFS*
> *2014-01-08 18:41:02,569 [IPC Server handler 24 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|dedb0fec255422*
> *f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018.34f3ef516fe6fba940eeb0902b9acd3d.,
> src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789,
> d*
> *est=inspur254.deu.edu.cn
> <http://inspur254.deu.edu.cn>,60020,1388054917364, running balancer*
> *2014-01-08 18:41:02,873 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|dedb0*
> *fec255422f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018.34f3ef516fe6fba940eeb0902b9acd3d.
> that was online on inspur254.deu.edu.cn <http://inspur254.deu.edu.cn>*
> *,60020,1388054917364*
> *2014-01-08 18:41:04,184 [IPC Server handler 44 on 60000] INFO
> org.apache.hadoop.hbase.master.HMaster - Added move plan
> hri=test,|e4e49102c1e6ea*
> *97094a40c57420a628|1381085596723|42696720858381436,1382457119052.ac201a56d80f13ca5357d474578a91c2.,
> src=inspur251.deu.edu.cn
> <http://inspur251.deu.edu.cn>,60020,1388050383789, *
> *dest=inspur308.deu.edu.cn
> <http://inspur308.deu.edu.cn>,60020,1388059770705, running balancer*
> *2014-01-08 18:41:04,863 [main-EventThread] INFO
> org.apache.hadoop.hbase.master.AssignmentManager - The master has opened
> the region test,|e4e49*
> *102c1e6ea97094a40c57420a628|1381085596723|42696720858381436,1382457119052.ac201a56d80f13ca5357d474578a91c2.
> that was online on inspur308.photo.163.or*
> *g,60020,1388059770705*
> *2014-01-08 18:43:46,735 [935521285@qtp-711761606-0] WARN
> org.apache.hadoop.conf.Configuration - fs.default.name
> <http://fs.default.name> is deprecated. Instead, use fs.defau*
> *ltFS*
> *2014-01-08 18:46:51,704 [935521285@qtp-711761606-0] WARN
> org.apache.hadoop.conf.Configuration - fs.default.name
> <http://fs.default.name> is deprecated. Instead, use fs.defau*
> *ltFS*
> *2014-01-08 18:47:03,294 [935521285@qtp-711761606-0] INFO
> org.apache.zookeeper.ZooKeeper - Initiating client connection,
> connectString=inspur254.phot*
> *o.163.org <http://o.163.org>:2181,inspur253.deu.edu.cn
> <http://inspur253.deu.edu.cn>:2181,inspur252.deu.edu.cn
> <http://inspur252.deu.edu.cn>:2181,inspur251.deu.edu.cn
> <http://inspur251.deu.edu.cn>:2181,inspur255.deu.edu.cn
> <http://inspur255.deu.edu.cn>:2181 sessionTimeout=120*
> *0000
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@172b29ed*
> *2014-01-08 18:47:03,295 [935521285@qtp-711761606-0] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - The identifier of
> this process is *
> *25086@inspur249.deu.edu.cn <25...@inspur249.deu.edu.cn>*
> *2014-01-08 18:47:03,295
> [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn
> <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn
> - Opening socket c*
> *onnection to server inspur251.deu.edu.cn/172.17.7.1:2181
> <http://inspur251.deu.edu.cn/172.17.7.1:2181>. Will not attempt to
> authenticate using SASL (Unable to locate a login configuration)*
> *2014-01-08 18:47:03,297
> [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn
> <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn
> - Socket connectio*
> *n established to inspur251.deu.edu.cn/172.17.7.1:2181
> <http://inspur251.deu.edu.cn/172.17.7.1:2181>, initiating session*
> *2014-01-08 18:47:03,302
> [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn
> <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn
> - Session establishment complete on server
> inspur251.deu.edu.cn/172.17.7.1:2181
> <http://inspur251.deu.edu.cn/172.17.7.1:2181>, sessionid =
> 0x42b69781e0f11d, negotiated timeout = 300000*
> *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0] INFO
> org.apache.zookeeper.ZooKeeper - Session: 0x42b69781e0f11d closed*
> *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0-EventThread] INFO
> org.apache.zookeeper.ClientCnxn - EventThread shut down*
> *2014-01-08 18:58:19,346 [IPC Reader 6 on port 60000] WARN
> org.apache.hadoop.ipc.HBaseServer - Incorrect header or version mismatch
> from 172.17.4.249:54719 <http://172.17.4.249:54719> got version 4 expected
> version 3*
>
> *I killed the process around **2014-01-08 18:55.T**he hanging region (*
> 812912be704946d24c5f1b5e3184b2f5*) has not any log.*
>
> *Thanks*
> 在 2014年1月8日，23:58，Ted Yu <yu...@gmail.com> 写道：
>
> Can you pastebin master log around 2014-01-08 18:40 ?
>
> Thanks
>
>
> On Wed, Jan 8, 2014 at 3:57 AM, hzwangxx <wh...@163.com> wrote:
>
>> Hi, all
>> I restart a region server by using graceful_stop.sh
>> (bin/graceful_stop.sh --restart --reload --debug hostname), when running a
>> moment, the process hanging as follows:
>>
>> 2014-01-08 18:40:48,150 [main] INFO region_mover - Moving region
>> 78c953d53f6498664d9a067701a7e7d7 (42 of 340) to server=
>> inspur255.deu.edu.cn,60020,1388056934052
>> 2014-01-08 18:40:50,097 [main] INFO region_mover - Moving region
>> c621b3bf29262ca5248c03a8d6ebb41e (43 of 340) to server=
>> inspur253.deu.edu.cn,60020,1388053123213
>> 2014-01-08 18:40:51,652 [main] INFO region_mover - Moving region
>> 4dad873a6af4d3a9809339281c3cb34c (44 of 340) to server=
>> inspur254.deu.edu.cn,60020,1388054917364
>> 2014-01-08 18:40:56,701 [main] INFO region_mover - Moving region
>> 0e311941f5ff202bcefe57aa4079a188 (45 of 340) to server=
>> inspur253.deu.edu.cn,60020,1388053123213
>> 2014-01-08 18:40:58,632 [main] INFO region_mover - Moving region
>> 3a955559fb65caf32a05e18b8b6b93f8 (46 of 340) to server=
>> inspur308.deu.edu.cn,60020,1388059770705
>> 2014-01-08 18:41:02,127 [main] INFO region_mover - Moving region
>> 34f3ef516fe6fba940eeb0902b9acd3d (47 of 340) to server=
>> inspur254.deu.edu.cn,60020,1388054917364
>> 2014-01-08 18:41:03,689 [main] INFO region_mover - Moving region
>> ac201a56d80f13ca5357d474578a91c2 (48 of 340) to server=
>> inspur308.deu.edu.cn,60020,1388059770705
>> 2014-01-08 18:41:05,669 [main] INFO region_mover - Moving region
>> 812912be704946d24c5f1b5e3184b2f5 (49 of 340) to server=
>> inspur253.deu.edu.cn,60020,1388053123213
>>
>> I run ‘du' command to check the last region , which has not any data.
>> hadoop@inspur249:~/hbase$ hdfs dfs -du -s -h
>> /hbase/test/812912be704946d24c5f1b5e3184b2f5/*
>> 486 /hbase/test/812912be704946d24c5f1b5e3184b2f5/.regioninfo
>> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/body
>> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/meta
>>
>> hadoop version is cdh4.2.1 and hbase is 0.94
>>
>> Thanks!
>> Best Regards~
>> Xiyi
>>
>>
>
>