You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yi Liang <wh...@gmail.com> on 2011/08/01 09:33:00 UTC
Can't disable a table because of a region time out
We're running hbase 0.90.3. For some unknown reason, we now can't disable
one table because its first region can't be unassigned.
The log message looks like following and it repeats endlessly:
2011-07-25 13:27:23,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571458790
2011-07-25 13:27:23,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-07-25 13:27:23,746 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-07-25 13:27:23,746 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-07-25 13:27:23,748 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 1 region(s) in transition:
{aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571643746}
2011-07-25 13:28:55,308 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server /192.168.201.7:2181
2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to test07.xxx.com/192.168.201.7:2181, initiating
session
2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server test07.xxx.com/192.168.201.7:2181,
sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
2011-07-25 13:30:17,583 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
max=2147483647 rows
2011-07-25 13:30:17,627 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x3314bcb3ae38799
2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
0x3314bcb3ae38799 closed
2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-07-25 13:30:33,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1311571643746
2011-07-25 13:30:33,745 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
The problem also blocks the region balancer.
Can I fix it without restarting the whole hbase cluster?
Thanks,
Yi
Re: Can't disable a table because of a region time out
Posted by Yi Liang <wh...@gmail.com>.
For people who don't want to restart the whole cluster, I have solved the
problem by restarting master alone and manually cleaning the table's ZK
state.
Thanks Jia for the suggestion of restarting master alone.
Thanks,
Yi
On Fri, Aug 5, 2011 at 5:35 PM, Yi Liang <wh...@gmail.com> wrote:
> Looks like the issue of HBASE-4064.
>
> Hope it will be fixed ASAP. But for now, how can I fix the issue?
>
> Due to the region in transition forever, the hbase balancer doesn't work
> any more.
>
> I have tried to kill and restart master alone, but it doesn't help. Do we
> have to restart the whole cluster for this region?
>
> Thanks,
> Yi
>
> On Thu, Aug 4, 2011 at 10:07 AM, Yi Liang <wh...@gmail.com> wrote:
>
>> HI J-D,
>>
>> I have tried to force unassign it with shell command 'unassign
>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.', true',
>> but it didn't help.
>>
>> The error message still appears after executing that command, and it
>> looks like just same as before:
>> 2011-08-04 09:56:41,005 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e.
>> (offlining)
>> 2011-08-04 09:56:41,005 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
>> region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. but
>> it is not currently assigned anywhere
>> 2011-08-04 09:57:09,847 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> state=PENDING_CLOSE, ts=1312422839848
>> 2011-08-04 09:57:09,848 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> PENDING_CLOSE for too long, running forced unassign again on
>> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> 2011-08-04 09:57:09,849 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> (offlining)
>> 2011-08-04 09:57:09,850 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> which is already pending close but forcing an additional close
>> 2011-08-04 09:57:09,851 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
>> test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
>> usedHeap=11121, maxHeap=15983) returned
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
>> not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
>> 2011-08-04 09:57:49,089 INFO
>> org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1
>> regions.
>> 2011-08-04 09:58:53,791 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
>> running balancer because 1 region(s) in transition:
>> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> state=PENDING_CLOSE, ts=1312423029850}
>> 2011-08-04 09:58:55,724 DEBUG
>> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5422 catalog row(s)
>> and gc'd 0 unreferenced parent region(s)
>> 2011-08-04 10:00:19,848 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out:
>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> state=PENDING_CLOSE, ts=1312423029850
>> 2011-08-04 10:00:19,848 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> PENDING_CLOSE for too long, running forced unassign again on
>> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> 2011-08-04 10:00:19,850 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
>> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> (offlining)
>> 2011-08-04 10:00:19,850 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> which is already pending close but forcing an additional close
>> 2011-08-04 10:00:19,852 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
>> test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
>> usedHeap=11121, maxHeap=15983) returned
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
>> not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
>>
>> Let me know if I have made any mistake.
>>
>> Thanks,
>> Yi
>>
>> On Tue, Aug 2, 2011 at 4:38 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>>
>>> You need to force unassign it using the shell.
>>>
>>> J-D
>>>
>>> On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <wh...@gmail.com> wrote:
>>> > We're running hbase 0.90.3. For some unknown reason, we now can't
>>> disable
>>> > one table because its first region can't be unassigned.
>>> >
>>> > The log message looks like following and it repeats endlessly:
>>> >
>>> > 2011-07-25 13:27:23,745 INFO
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>>> > timed out:
>>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > state=PENDING_CLOSE, ts=1311571458790
>>> > 2011-07-25 13:27:23,745 INFO
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>>> > PENDING_CLOSE for too long, running forced unassign again on
>>> >
>>> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > 2011-07-25 13:27:23,746 DEBUG
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment
>>> of
>>> > region
>>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > (offlining)
>>> > 2011-07-25 13:27:23,746 DEBUG
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to
>>> unassign
>>> > region
>>> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > which is already pending close but forcing an additional close
>>> > 2011-07-25 13:27:23,748 DEBUG
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
>>> > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
>>> > usedHeap=11121, maxHeap=15983) returned
>>> > org.apache.hadoop.hbase.NotServingRegionException:
>>> > org.apache.hadoop.hbase.NotServingRegionException: Received close for
>>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but
>>> we are
>>> > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
>>> > 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster:
>>> Not
>>> > running balancer because 1 region(s) in transition:
>>> >
>>> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > state=PENDING_CLOSE, ts=1311571643746}
>>> > 2011-07-25 13:28:55,308 DEBUG
>>> org.apache.hadoop.hbase.master.CatalogJanitor:
>>> > Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
>>> > 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>> > client connection, connectString=192.168.201.7:2181,192.168.201.6:2181
>>> ,
>>> > 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
>>> > 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket
>>> > connection to server /192.168.201.7:2181
>>> > 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
>>> > connection established to test07.xxx.com/192.168.201.7:2181,
>>> initiating
>>> > session
>>> > 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
>>> > establishment complete on server test07.xxx.com/192.168.201.7:2181,
>>> > sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
>>> > 2011-07-25 13:30:17,583 DEBUG
>>> org.apache.hadoop.hbase.client.MetaScanner:
>>> > Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
>>> > max=2147483647 rows
>>> > 2011-07-25 13:30:17,627 INFO
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>>> > Closed zookeeper sessionid=0x3314bcb3ae38799
>>> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
>>> > 0x3314bcb3ae38799 closed
>>> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn:
>>> EventThread
>>> > shut down
>>> > 2011-07-25 13:30:33,745 INFO
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>>> > timed out:
>>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> > state=PENDING_CLOSE, ts=1311571643746
>>> > 2011-07-25 13:30:33,745 INFO
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>>> > PENDING_CLOSE for too long, running forced unassign again on
>>> >
>>> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>>> >
>>> > The problem also blocks the region balancer.
>>> >
>>> > Can I fix it without restarting the whole hbase cluster?
>>> >
>>> > Thanks,
>>> > Yi
>>> >
>>>
>>
>>
>
Re: Can't disable a table because of a region time out
Posted by Yi Liang <wh...@gmail.com>.
Looks like the issue of HBASE-4064.
Hope it will be fixed ASAP. But for now, how can I fix the issue?
Due to the region in transition forever, the hbase balancer doesn't work any
more.
I have tried to kill and restart master alone, but it doesn't help. Do we
have to restart the whole cluster for this region?
Thanks,
Yi
On Thu, Aug 4, 2011 at 10:07 AM, Yi Liang <wh...@gmail.com> wrote:
> HI J-D,
>
> I have tried to force unassign it with shell command 'unassign
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.', true',
> but it didn't help.
>
> The error message still appears after executing that command, and it looks
> like just same as before:
> 2011-08-04 09:56:41,005 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
> region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e.
> (offlining)
> 2011-08-04 09:56:41,005 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
> region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. but
> it is not currently assigned anywhere
> 2011-08-04 09:57:09,847 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> timed out:
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1312422839848
> 2011-08-04 09:57:09,848 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> PENDING_CLOSE for too long, running forced unassign again on
> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> 2011-08-04 09:57:09,849 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> (offlining)
> 2011-08-04 09:57:09,850 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> which is already pending close but forcing an additional close
> 2011-08-04 09:57:09,851 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
> test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
> usedHeap=11121, maxHeap=15983) returned
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Received close for
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
> not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
> 2011-08-04 09:57:49,089 INFO
> org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1
> regions.
> 2011-08-04 09:58:53,791 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
> running balancer because 1 region(s) in transition:
> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1312423029850}
> 2011-08-04 09:58:55,724 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5422 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2011-08-04 10:00:19,848 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> timed out:
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1312423029850
> 2011-08-04 10:00:19,848 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> PENDING_CLOSE for too long, running forced unassign again on
> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> 2011-08-04 10:00:19,850 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> (offlining)
> 2011-08-04 10:00:19,850 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> which is already pending close but forcing an additional close
> 2011-08-04 10:00:19,852 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
> test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
> usedHeap=11121, maxHeap=15983) returned
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Received close for
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
> not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
>
> Let me know if I have made any mistake.
>
> Thanks,
> Yi
>
> On Tue, Aug 2, 2011 at 4:38 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> You need to force unassign it using the shell.
>>
>> J-D
>>
>> On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <wh...@gmail.com> wrote:
>> > We're running hbase 0.90.3. For some unknown reason, we now can't
>> disable
>> > one table because its first region can't be unassigned.
>> >
>> > The log message looks like following and it repeats endlessly:
>> >
>> > 2011-07-25 13:27:23,745 INFO
>> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> > timed out:
>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > state=PENDING_CLOSE, ts=1311571458790
>> > 2011-07-25 13:27:23,745 INFO
>> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> > PENDING_CLOSE for too long, running forced unassign again on
>> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > 2011-07-25 13:27:23,746 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment
>> of
>> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > (offlining)
>> > 2011-07-25 13:27:23,746 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
>> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > which is already pending close but forcing an additional close
>> > 2011-07-25 13:27:23,748 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
>> > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
>> > usedHeap=11121, maxHeap=15983) returned
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> > org.apache.hadoop.hbase.NotServingRegionException: Received close for
>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we
>> are
>> > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
>> > 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster:
>> Not
>> > running balancer because 1 region(s) in transition:
>> >
>> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > state=PENDING_CLOSE, ts=1311571643746}
>> > 2011-07-25 13:28:55,308 DEBUG
>> org.apache.hadoop.hbase.master.CatalogJanitor:
>> > Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
>> > 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> > client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
>> > 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
>> > 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket
>> > connection to server /192.168.201.7:2181
>> > 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
>> > connection established to test07.xxx.com/192.168.201.7:2181, initiating
>> > session
>> > 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
>> > establishment complete on server test07.xxx.com/192.168.201.7:2181,
>> > sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
>> > 2011-07-25 13:30:17,583 DEBUG
>> org.apache.hadoop.hbase.client.MetaScanner:
>> > Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
>> > max=2147483647 rows
>> > 2011-07-25 13:30:17,627 INFO
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Closed zookeeper sessionid=0x3314bcb3ae38799
>> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x3314bcb3ae38799 closed
>> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn:
>> EventThread
>> > shut down
>> > 2011-07-25 13:30:33,745 INFO
>> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> > timed out:
>> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> > state=PENDING_CLOSE, ts=1311571643746
>> > 2011-07-25 13:30:33,745 INFO
>> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> > PENDING_CLOSE for too long, running forced unassign again on
>> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>> >
>> > The problem also blocks the region balancer.
>> >
>> > Can I fix it without restarting the whole hbase cluster?
>> >
>> > Thanks,
>> > Yi
>> >
>>
>
>
Re: Can't disable a table because of a region time out
Posted by Yi Liang <wh...@gmail.com>.
HI J-D,
I have tried to force unassign it with shell command 'unassign
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.', true',
but it didn't help.
The error message still appears after executing that command, and it looks
like just same as before:
2011-08-04 09:56:41,005 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e.
(offlining)
2011-08-04 09:56:41,005 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
region HistoryNoticeInc,,1311313783815.f2ece755144ab673757896c0c374783e. but
it is not currently assigned anywhere
2011-08-04 09:57:09,847 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312422839848
2011-08-04 09:57:09,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-08-04 09:57:09,849 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-08-04 09:57:09,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-08-04 09:57:09,851 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
2011-08-04 09:57:49,089 INFO
org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1
regions.
2011-08-04 09:58:53,791 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
running balancer because 1 region(s) in transition:
{aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312423029850}
2011-08-04 09:58:55,724 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 5422 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-08-04 10:00:19,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
timed out:
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
state=PENDING_CLOSE, ts=1312423029850
2011-08-04 10:00:19,848 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
2011-08-04 10:00:19,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
(offlining)
2011-08-04 10:00:19,850 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
which is already pending close but forcing an additional close
2011-08-04 10:00:19,852 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
usedHeap=11121, maxHeap=15983) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
Let me know if I have made any mistake.
Thanks,
Yi
On Tue, Aug 2, 2011 at 4:38 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> You need to force unassign it using the shell.
>
> J-D
>
> On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <wh...@gmail.com> wrote:
> > We're running hbase 0.90.3. For some unknown reason, we now can't disable
> > one table because its first region can't be unassigned.
> >
> > The log message looks like following and it repeats endlessly:
> >
> > 2011-07-25 13:27:23,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> > timed out:
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571458790
> > 2011-07-25 13:27:23,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> > PENDING_CLOSE for too long, running forced unassign again on
> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > 2011-07-25 13:27:23,746 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment
> of
> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > (offlining)
> > 2011-07-25 13:27:23,746 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
> > region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > which is already pending close but forcing an additional close
> > 2011-07-25 13:27:23,748 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
> > test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
> > usedHeap=11121, maxHeap=15983) returned
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.NotServingRegionException: Received close for
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we
> are
> > not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
> > 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
> > running balancer because 1 region(s) in transition:
> >
> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571643746}
> > 2011-07-25 13:28:55,308 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor:
> > Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
> > 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
> > client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
> > 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
> > 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> > connection to server /192.168.201.7:2181
> > 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to test07.xxx.com/192.168.201.7:2181, initiating
> > session
> > 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
> > establishment complete on server test07.xxx.com/192.168.201.7:2181,
> > sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
> > 2011-07-25 13:30:17,583 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> > Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
> > max=2147483647 rows
> > 2011-07-25 13:30:17,627 INFO
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Closed zookeeper sessionid=0x3314bcb3ae38799
> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x3314bcb3ae38799 closed
> > 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: EventThread
> > shut down
> > 2011-07-25 13:30:33,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> > timed out:
> > HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> > state=PENDING_CLOSE, ts=1311571643746
> > 2011-07-25 13:30:33,745 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> > PENDING_CLOSE for too long, running forced unassign again on
> > region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> >
> > The problem also blocks the region balancer.
> >
> > Can I fix it without restarting the whole hbase cluster?
> >
> > Thanks,
> > Yi
> >
>
Re: Can't disable a table because of a region time out
Posted by Jean-Daniel Cryans <jd...@apache.org>.
You need to force unassign it using the shell.
J-D
On Mon, Aug 1, 2011 at 12:33 AM, Yi Liang <wh...@gmail.com> wrote:
> We're running hbase 0.90.3. For some unknown reason, we now can't disable
> one table because its first region can't be unassigned.
>
> The log message looks like following and it repeats endlessly:
>
> 2011-07-25 13:27:23,745 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> timed out:
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1311571458790
> 2011-07-25 13:27:23,745 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> PENDING_CLOSE for too long, running forced unassign again on
> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> 2011-07-25 13:27:23,746 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> (offlining)
> 2011-07-25 13:27:23,746 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Attempting to unassign
> region HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> which is already pending close but forcing an additional close
> 2011-07-25 13:27:23,748 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Server serverName=
> test28.xxx.com,60020,1309749632386, load=(requests=0, regions=237,
> usedHeap=11121, maxHeap=15983) returned
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Received close for
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa. but we are
> not serving it for aaa8d345f5b7b6a69b786fe6d14ed9fa
> 2011-07-25 13:28:53,508 DEBUG org.apache.hadoop.hbase.master.HMaster: Not
> running balancer because 1 region(s) in transition:
> {aaa8d345f5b7b6a69b786fe6d14ed9fa=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1311571643746}
> 2011-07-25 13:28:55,308 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
> Scanned 5424 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-07-25 13:30:17,565 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=192.168.201.7:2181,192.168.201.6:2181,
> 192.168.201.5:2181 sessionTimeout=60000 watcher=hconnection
> 2011-07-25 13:30:17,566 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server /192.168.201.7:2181
> 2011-07-25 13:30:17,568 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to test07.xxx.com/192.168.201.7:2181, initiating
> session
> 2011-07-25 13:30:17,572 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server test07.xxx.com/192.168.201.7:2181,
> sessionid = 0x3314bcb3ae38799, negotiated timeout = 60000
> 2011-07-25 13:30:17,583 DEBUG org.apache.hadoop.hbase.client.MetaScanner:
> Scanning .META. starting at row=HistoryNoticeInc,,00000000000000 for
> max=2147483647 rows
> 2011-07-25 13:30:17,627 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Closed zookeeper sessionid=0x3314bcb3ae38799
> 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x3314bcb3ae38799 closed
> 2011-07-25 13:30:17,637 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-07-25 13:30:33,745 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> timed out:
> HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
> state=PENDING_CLOSE, ts=1311571643746
> 2011-07-25 13:30:33,745 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
> PENDING_CLOSE for too long, running forced unassign again on
> region=HistoryNoticeInc,,1311223940614.aaa8d345f5b7b6a69b786fe6d14ed9fa.
>
> The problem also blocks the region balancer.
>
> Can I fix it without restarting the whole hbase cluster?
>
> Thanks,
> Yi
>