You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Laurent Chavet (JIRA)" <ji...@apache.org> on 2011/04/20 23:08:06 UTC
[jira] [Created] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
-----------------------------------------------------------------------------
Key: HBASE-3804
URL: https://issues.apache.org/jira/browse/HBASE-3804
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.2
Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
Single node cluster
Reporter: Laurent Chavet
Priority: Blocker
After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
015a9f9c03a3
2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
6f486919c9b5
2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305143661
2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305152287
2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305160979
2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305170430
2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305178662
2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305188551
2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305196703
2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305205039
2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305213412
2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305220665
2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305231697
2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305239174
2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305248013
2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305256411
2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305265663
2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305274667
2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305283140
2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305291307
2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305300632
2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305309439
2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305316405
2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305322730
2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305330453
2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305337020
2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305345309
2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305352766
2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305362335
2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305370474
2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305379360
2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305387434
2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305396557
2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305403848
2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305412950
2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305420349
2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305429358
2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305437383
2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305446715
2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305460169
2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305467377
2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305476873
2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305484717
2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305492251
2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
ldlogs/HQSNOSQL01%3A60020.1303305500805
2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
finish
2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
eper
2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
dating) unassigned node for 70236052 with OFFLINE state
2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
dating) unassigned node for 1028785192 with OFFLINE state
2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
server=HQSNOSQL01:60000, region=1028785192/.META.
2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305598133
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305598113
2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305598133
2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305598113
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305636364
2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305636364
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305666374
2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305666374
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305696385
2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305696385
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305726393
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305726394
2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305726393
2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305726394
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305756402
2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305756402
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
2 state=OFFLINE, ts=1303305786412
2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
state=OFFLINE, ts=1303305786412
2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
,1.1028785192 state=OFFLINE, ts=1303305816420
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning .META.,,1.1028785192 to a random server
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
,0.70236052 state=OFFLINE, ts=1303305816420
2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
ssigning -ROOT-,,0.70236052 to a random server
2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022831#comment-13022831 ]
Jean-Daniel Cryans commented on HBASE-3804:
-------------------------------------------
So how it works: for some reason your region server died. A region server is what carries and serves the regions (so the data). One thing that's not clear to me is if you are running in "standalone" mode or "pseudo-distributed" mode. If the former, then yeah I agree that it's a bit dumb that it locks up like that because you cannot restart the region server _inside_ the master process. If you were using the latter, then well your region server died and you need to restart it.
What the master does when a node dies is that it splits the write ahead logs (that's what's going on in the beginning of your log) and reassigns the regions. In your case, there's no one to reassign the regions to. The master cannot carry the regions, so it's waiting for a new region server to come in.
On to why the region server died, there should be a hint in the log. Something on the FATAL log level. Probably GC pauses.
BTW the book has this to say on single node clusters: "Do not use this configuration for production nor for evaluating HBase performance."
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
Posted by "Laurent Chavet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022842#comment-13022842 ]
Laurent Chavet commented on HBASE-3804:
---------------------------------------
Thanks very much; that was it.
I am running in "pseudo-distributed" and the region server did die (I need to increase the limit on open files).
It all makes sense to me now; thanks for the very fast answers.
BTW: is there standard way to automatically restart the region severs when they crash or should I write my own small script to deal with that.
Actually do you even think it's a good idea to automatically restart (I might have missed the real problem)?
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
Posted by "Laurent Chavet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Laurent Chavet resolved HBASE-3804.
-----------------------------------
Resolution: Not A Problem
see comments
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
Posted by "Laurent Chavet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022828#comment-13022828 ]
Laurent Chavet commented on HBASE-3804:
---------------------------------------
This sounds like good news; it sounds like the problem would not exist once I move my project to a multi nodes cluster.
Yet I am surprised that this behavior where hbase completely looks up would be expected (even in a single node cluster); can you explain more what is going on and why the single node cluster can't cope with it?
Thanks
Laurent Chavet
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3804) HBase gets stuck with "Regions in
transition timed out" for .META. and -ROOT-
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022383#comment-13022383 ]
Jean-Daniel Cryans commented on HBASE-3804:
-------------------------------------------
So you have one node? And the region server's session expired? If so, then that's the expected behavior because the master cannot assign those regions.
> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
> Key: HBASE-3804
> URL: https://issues.apache.org/jira/browse/HBASE-3804
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.2
> Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
> Reporter: Laurent Chavet
> Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively 50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't recover without a restart (I copy a large chunk of the master logs before the problem in case it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251, length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805, length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805 to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote 134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote 30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote 22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote 85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote 117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote 63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote 53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote 134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote 77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote 10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote 117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote 31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote 71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote 77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote 24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote 8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote 5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote 125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote 1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote 1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote 53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote 155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote 53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote 1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote 96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000 Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE,
> server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=.META.,,1.102878519
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira