You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by bijieshan <bi...@huawei.com> on 2011/05/31 13:41:33 UTC
A sudden msg of "java.io.IOException: Server not running, aborting"
It occurred in an RegionServer with an un-known reason. I have check this RegionServer logs, there's no prev-aborting, and no other infos showed the RegionServer has aborted. So I saw the following msg showed in a sudden.
>>[logs]
2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>[description]
Before that time, everything seemed ok(it received so many open-region requests, and handled them), except several Exception happened during the time.
>>[logs]
2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,110018,1305617626846.f22eafaf06c3be7f7e5a447cec40c7fa.
2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,111336,1305617636659.b47f1cf09b10aecaee6805a985646b30.
2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280884,1305617952747.7cac8ccc9c9770ef0e48807e07bc8bec.
2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280446,1305617949439.b332cfff565d77b2e2d4f1a2c52494b0.
2011-05-25 09:15:36,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,101974,1305617623149.b60c71565f2c73b81477fe25ba80378b.
2011-05-25 09:15:36,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:36,423 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:37,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:37,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:37,660 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:37,662 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:37,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>>[description]
>From the code, I saw that , only the Server has aborted, the msg of "Server not running, aborting" could be shown. It determined by the param of HRegionServer#abortRequested, if it was true, and if somewhere called the method of HRegionServer#checkOpen, this msg would been printed.
/**
* Called to verify that this server is up and running.
*
* @throws IOException
*/
protected void checkOpen() throws IOException {
if (this.stopped || this.abortRequested) {
throw new IOException("Server not running"
+ (this.abortRequested ? ", aborting" : ""));
}
if (!fsOk) {
throw new IOException("File system not available");
}
}
For I said there's no aborted info been found in Logs:
public void abort(String reason, Throwable cause) {
if (cause != null) {
LOG.fatal("ABORTING region server " + this + ": " + reason, cause);
} else {
LOG.fatal("ABORTING region server " + this + ": " + reason);
}
this.abortRequested = true;
this.reservedSpace.clear();
if (this.metrics != null) {
LOG.info("Dump of metrics: " + this.metrics);
}
stop(reason);
}
For there's no prev-aborting found in the logs, how did the sudden msg of " java.io.IOException: Server not running, aborting " occurred?
Thanks!
Jieshan Bean.
Re: Same Region could be picked out twice in LoadBalancer
Posted by Stack <st...@duboce.net>.
The below seems right. Do you have a patch Jieshan?
St.Ack
On Sun, Jun 12, 2011 at 2:41 AM, bijieshan <bi...@huawei.com> wrote:
> From the HMaster logs, I found something weird:
>
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
>
> We can see that, the same region was balanced twice.
>
> To describe the problem, I give out one simple example:
>
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
>
> Here's the analysis from code:
>
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
>
> If I make any mistakes in the analysis, please point them out.
> Thanks!
>
> Jieshan Bean
>
>
Same Region could be picked out twice in LoadBalancer
Posted by bijieshan <bi...@huawei.com>.
>From the HMaster logs, I found something weird:
2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
We can see that, the same region was balanced twice.
To describe the problem, I give out one simple example:
1. Suppose regions count is 10 in RegionServer A.
Max: 5 Min:4
2. So the regions count need to move is: 5.
3. Before the movement of calculate, the list was shuffled.
4. The 5 moving region was picked out from the back.
5. The nextRegionForUnload value is 5.
6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
This time , the picked Index is 5 which has been picked once!!!!!
|<-----5-------|
------------*--*--*--*--*--*--*--*--*--*----
|
getNextRegionForUnload
Here's the analysis from code:
1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
Map<HServerInfo,BalanceInfo> serverBalanceInfo =
new TreeMap<HServerInfo,BalanceInfo>();
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
HServerInfo serverInfo = server.getKey();
int regionCount = serverInfo.getLoad().getNumberOfRegions();
if(regionCount <= max) {
serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
break;
}
serversOverloaded++;
List<HRegionInfo> regions = randomize(server.getValue());
int numToOffload = Math.min(regionCount - max, regions.size());
int numTaken = 0;
for (int i = regions.size() - 1; i >= 0; i--) {
HRegionInfo hri = regions.get(i);
// Don't rebalance meta regions.
if (hri.isMetaRegion()) continue;
regionsToMove.add(new RegionPlan(hri, serverInfo, null));
numTaken++;
if (numTaken >= numToOffload) break;
}
/**********************************************************/
/***set the nextRegionForUnload value is numToOffload ****/
/**********************************************************/
serverBalanceInfo.put(serverInfo,
new BalanceInfo(numToOffload, (-1)*numTaken));
}
2. The second pass of picked one region from the Max regionserver by order.
if (neededRegions != 0) {
// Walk down most loaded, grabbing one from each until we get enough
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
int idx =
balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
if (idx >= server.getValue().size()) break;
HRegionInfo region = server.getValue().get(idx);
if (region.isMetaRegion()) continue; // Don't move meta regions.
regionsToMove.add(new RegionPlan(region, server.getKey(), null));
if(--neededRegions == 0) {
// No more regions needed, done shedding
break;
}
}
}
If I make any mistakes in the analysis, please point them out.
Thanks!
Jieshan Bean
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Awesome, this is a very good bug. I filed
https://issues.apache.org/jira/browse/HBASE-3984
J-D
On Sun, Jun 12, 2011 at 2:03 AM, bijieshan <bi...@huawei.com> wrote:
> Thanks J-D.
> I have got the reasons.
> The original .META. location is 100-9. So after verify .META. region location and failed, the location will be reset.
>
> But while verifying, a Exception occurred:
>
> 100-9 was happening to OOM, so the calling of verifyRegionLocaton was blocked. After some time, the thread was interrupted. And a ClosedByInterruptedException occurred.
>
> So the OOM is the main cause of the problem.
>
> Jieshan Bean
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by bijieshan <bi...@huawei.com>.
Thanks J-D.
I have got the reasons.
The original .META. location is 100-9. So after verify .META. region location and failed, the location will be reset.
But while verifying, a Exception occurred:
100-9 was happening to OOM, so the calling of verifyRegionLocaton was blocked. After some time, the thread was interrupted. And a ClosedByInterruptedException occurred.
So the OOM is the main cause of the problem.
Jieshan Bean
----------------------
Ah so 100-13 was the .META. holder but got a socket timeout trying to
talk to 100-9 which was the previous .META. holder.
It means that when it goes to do a put at 09:15:44,232 it must be
contacting that same region server since the location wasn't updated
(the first call failed on socket timeout). Can you check what's going
on with 100-9 during that time and if it's really shutting down?
J-D
On Tue, Jun 7, 2011 at 9:23 AM, bijieshan <bi...@huawei.com> wrote:
> Thanks J-D.
> Sorry for a long time break of the reply!
> I check the logs of the .META. regionserver, it's indeed the problem like you described.
> But I found another problem.
>
> The .META. Region has changed it's address, but last for a long time, CatalogTracker still cache the old address.
> So while another regionserver(not the .META. regionserver) split region, it will send IPC request to put, this will execute in the old regionserver.
>
>>>From HMaster, we can see that at 09:02:34, the region has been opened in 100-13:
>
> 2011-05-25 08:37:03,908 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-9,20020,1306257984044
> 2011-05-25 09:02:34,334 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-13,20020,1306266506022
> 2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=157-5-100-9:20020; java.io.EOFException
> 2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Current cached META location is not valid, resetting
>
>>>From RegionServer 100-13 and at 09:15:44, we can see the .META. address cached in CatalogTracker was still 100-9
>
> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> 2011-05-25 09:15:44,232 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=11dc72d94c7a5a3d19b0c0c3c49624a5
> java.io.IOException: Call to 157-5-100-9/157.5.100.9:20020 failed on local exception: java.nio.channels.ClosedByInterruptException
> at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy8.getRegionInfo(Unknown Source)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
> Caused by: java.nio.channels.ClosedByInterruptException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
> ... 9 more
>
>
> Is it a bug here? I think the cached info should be invalid and reset after the .META. address has been changed immediately , but it's not.
>
> Thanks!
>
> Jieshan Bean
>
>
>
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Ah so 100-13 was the .META. holder but got a socket timeout trying to
talk to 100-9 which was the previous .META. holder.
It means that when it goes to do a put at 09:15:44,232 it must be
contacting that same region server since the location wasn't updated
(the first call failed on socket timeout). Can you check what's going
on with 100-9 during that time and if it's really shutting down?
J-D
On Tue, Jun 7, 2011 at 9:23 AM, bijieshan <bi...@huawei.com> wrote:
> Thanks J-D.
> Sorry for a long time break of the reply!
> I check the logs of the .META. regionserver, it's indeed the problem like you described.
> But I found another problem.
>
> The .META. Region has changed it's address, but last for a long time, CatalogTracker still cache the old address.
> So while another regionserver(not the .META. regionserver) split region, it will send IPC request to put, this will execute in the old regionserver.
>
>>>From HMaster, we can see that at 09:02:34, the region has been opened in 100-13:
>
> 2011-05-25 08:37:03,908 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-9,20020,1306257984044
> 2011-05-25 09:02:34,334 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-13,20020,1306266506022
> 2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=157-5-100-9:20020; java.io.EOFException
> 2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Current cached META location is not valid, resetting
>
>>>From RegionServer 100-13 and at 09:15:44, we can see the .META. address cached in CatalogTracker was still 100-9
>
> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> 2011-05-25 09:15:44,232 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=11dc72d94c7a5a3d19b0c0c3c49624a5
> java.io.IOException: Call to 157-5-100-9/157.5.100.9:20020 failed on local exception: java.nio.channels.ClosedByInterruptException
> at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy8.getRegionInfo(Unknown Source)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
> Caused by: java.nio.channels.ClosedByInterruptException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
> ... 9 more
>
>
> Is it a bug here? I think the cached info should be invalid and reset after the .META. address has been changed immediately , but it's not.
>
> Thanks!
>
> Jieshan Bean
>
>
>
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by bijieshan <bi...@huawei.com>.
Thanks J-D.
Sorry for a long time break of the reply!
I check the logs of the .META. regionserver, it's indeed the problem like you described.
But I found another problem.
The .META. Region has changed it's address, but last for a long time, CatalogTracker still cache the old address.
So while another regionserver(not the .META. regionserver) split region, it will send IPC request to put, this will execute in the old regionserver.
>>From HMaster, we can see that at 09:02:34, the region has been opened in 100-13:
2011-05-25 08:37:03,908 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-9,20020,1306257984044
2011-05-25 09:02:34,334 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region .META.,,1.1028785192 on 157-5-100-13,20020,1306266506022
2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=157-5-100-9:20020; java.io.EOFException
2011-05-25 09:15:57,649 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Current cached META location is not valid, resetting
>>From RegionServer 100-13 and at 09:15:44, we can see the .META. address cached in CatalogTracker was still 100-9
2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
2011-05-25 09:15:44,232 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=11dc72d94c7a5a3d19b0c0c3c49624a5
java.io.IOException: Call to 157-5-100-9/157.5.100.9:20020 failed on local exception: java.nio.channels.ClosedByInterruptException
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
... 9 more
Is it a bug here? I think the cached info should be invalid and reset after the .META. address has been changed immediately , but it's not.
Thanks!
Jieshan Bean
-----------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
Oh I'm so dumb, I didn't see that this is coming from _another_ region
server. What happens when splitting is that we try to update the
.META. table which requires talking to another region server, except
that in this case the RS was aborting meaning that the split
transaction had to be rolled back.
Looking at the logs of the RS that carried .META. at that time would
give you more clues into what really happened.
J-D
On Tue, May 31, 2011 at 5:24 PM, bijieshan <bi...@huawei.com> wrote:
> I paste the logs as following. To short of the logs, some similar and closer logs I deleted:
>
>> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>>>>[description]
>> Before that time, everything seemed ok(it received so many open-region requests, and handled them), except several Exception happened during the time.
>>
>>>>[logs]
>> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,110018,1305617626846.f22eafaf06c3be7f7e5a447cec40c7fa.
>> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,111336,1305617636659.b47f1cf09b10aecaee6805a985646b30.
>> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280884,1305617952747.7cac8ccc9c9770ef0e48807e07bc8bec.
>> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280446,1305617949439.b332cfff565d77b2e2d4f1a2c52494b0.
>> 2011-05-25 09:15:36,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,101974,1305617623149.b60c71565f2c73b81477fe25ba80378b.
>> 2011-05-25 09:15:36,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:36,423 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,660 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,662 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>>
>>>>[description]
>> From the code, I saw that , only the Server has aborted, the msg of "Server not running, aborting" could be shown. It determined by the param of HRegionServer#abortRequested, if it was true, and if somewhere called the method of HRegionServer#checkOpen, this msg would been printed.
>>
>> /**
>> * Called to verify that this server is up and running.
>> *
>> * @throws IOException
>> */
>> protected void checkOpen() throws IOException {
>> if (this.stopped || this.abortRequested) {
>> throw new IOException("Server not running"
>> + (this.abortRequested ? ", aborting" : ""));
>> }
>> if (!fsOk) {
>> throw new IOException("File system not available");
>> }
>> }
>>
>> For I said there's no aborted info been found in Logs:
>>
>> public void abort(String reason, Throwable cause) {
>> if (cause != null) {
>> LOG.fatal("ABORTING region server " + this + ": " + reason, cause);
>> } else {
>> LOG.fatal("ABORTING region server " + this + ": " + reason);
>> }
>> this.abortRequested = true;
>> this.reservedSpace.clear();
>> if (this.metrics != null) {
>> LOG.info("Dump of metrics: " + this.metrics);
>> }
>> stop(reason);
>> }
>>
>> For there's no prev-aborting found in the logs, how did the sudden msg of " java.io.IOException: Server not running, aborting " occurred?
>> Thanks!
>>
>> Jieshan Bean.
>>
>>
>>
>>
>>
>
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Oh I'm so dumb, I didn't see that this is coming from _another_ region
server. What happens when splitting is that we try to update the
.META. table which requires talking to another region server, except
that in this case the RS was aborting meaning that the split
transaction had to be rolled back.
Looking at the logs of the RS that carried .META. at that time would
give you more clues into what really happened.
J-D
On Tue, May 31, 2011 at 5:24 PM, bijieshan <bi...@huawei.com> wrote:
> I paste the logs as following. To short of the logs, some similar and closer logs I deleted:
>
>> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>>>>[description]
>> Before that time, everything seemed ok(it received so many open-region requests, and handled them), except several Exception happened during the time.
>>
>>>>[logs]
>> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,110018,1305617626846.f22eafaf06c3be7f7e5a447cec40c7fa.
>> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,111336,1305617636659.b47f1cf09b10aecaee6805a985646b30.
>> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280884,1305617952747.7cac8ccc9c9770ef0e48807e07bc8bec.
>> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280446,1305617949439.b332cfff565d77b2e2d4f1a2c52494b0.
>> 2011-05-25 09:15:36,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,101974,1305617623149.b60c71565f2c73b81477fe25ba80378b.
>> 2011-05-25 09:15:36,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:36,423 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,660 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,662 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>> 2011-05-25 09:15:37,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>>
>>>>[description]
>> From the code, I saw that , only the Server has aborted, the msg of "Server not running, aborting" could be shown. It determined by the param of HRegionServer#abortRequested, if it was true, and if somewhere called the method of HRegionServer#checkOpen, this msg would been printed.
>>
>> /**
>> * Called to verify that this server is up and running.
>> *
>> * @throws IOException
>> */
>> protected void checkOpen() throws IOException {
>> if (this.stopped || this.abortRequested) {
>> throw new IOException("Server not running"
>> + (this.abortRequested ? ", aborting" : ""));
>> }
>> if (!fsOk) {
>> throw new IOException("File system not available");
>> }
>> }
>>
>> For I said there's no aborted info been found in Logs:
>>
>> public void abort(String reason, Throwable cause) {
>> if (cause != null) {
>> LOG.fatal("ABORTING region server " + this + ": " + reason, cause);
>> } else {
>> LOG.fatal("ABORTING region server " + this + ": " + reason);
>> }
>> this.abortRequested = true;
>> this.reservedSpace.clear();
>> if (this.metrics != null) {
>> LOG.info("Dump of metrics: " + this.metrics);
>> }
>> stop(reason);
>> }
>>
>> For there's no prev-aborting found in the logs, how did the sudden msg of " java.io.IOException: Server not running, aborting " occurred?
>> Thanks!
>>
>> Jieshan Bean.
>>
>>
>>
>>
>>
>
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by bijieshan <bi...@huawei.com>.
I paste the logs as following. To short of the logs, some similar and closer logs I deleted:
2011-05-25 09:14:56,305 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,042252,1305617508327.a0997be95a5259f04724640206844a55.
2011-05-25 09:14:56,305 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,260076,1305617910427.e4d3ba284a1fb369af122d28a659384a.
2011-05-25 09:14:56,305 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,250384,1305617894499.52f70b283354325070f6566c62d7f658.
2011-05-25 09:14:56,306 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,252110,1305617907665.77d9eca58fbfa84d44e1597ab7ce6638.
2011-05-25 09:14:56,308 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,042016,1305617506533.fbcd3c92982a789f41c359ac419b5bf0.
2011-05-25 09:14:56,308 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,050308,1305617511947.b5ed33832e75b3648d0ebd97586a20ac.
2011-05-25 09:14:59,616 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:14:59,634 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,776 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=3c207ea593f2b08157c6b444fa112154
java.net.SocketTimeoutException: Call to 157-5-100-9/157.5.100.9:20020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/157.5.100.13:34811 remote=157-5-100-9/157.5.100.9:20020]
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/157.5.100.13:34811 remote=157-5-100-9/157.5.100.9:20020]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:165)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:539)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
2011-05-25 09:15:00,777 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=53b0a08f4e583a46ad9676cafeb7d65f
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,777 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041544,1305617502930.53b0a08f4e583a46ad9676cafeb7d65f.: disabling compactions & flushes
2011-05-25 09:15:00,777 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041544,1305617502930.53b0a08f4e583a46ad9676cafeb7d65f.
2011-05-25 09:15:00,777 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041544,1305617502930.53b0a08f4e583a46ad9676cafeb7d65f.
2011-05-25 09:15:00,777 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=8ed760e2d9c762a6bda7e9f99f2cd340
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=eecfd47c62f16f324e734e831b039cfa
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,777 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=e77f1c70062156eb4dbba21e3a65ee07
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=044a911c7b6cc2c2e8583c7eda97f87e
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,777 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041476,1305617502415.3c207ea593f2b08157c6b444fa112154.: disabling compactions & flushes
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041476,1305617502415.3c207ea593f2b08157c6b444fa112154.
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,778 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041476,1305617502415.3c207ea593f2b08157c6b444fa112154.
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041624,1305617503522.c4c657deda864fa2018b38388ee928e6.
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=7b510ae3fc4047e3b33d33b9d1101fbc
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041422,1305617502025.e77f1c70062156eb4dbba21e3a65ee07.: disabling compactions & flushes
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041422,1305617502025.e77f1c70062156eb4dbba21e3a65ee07.
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=7a717d98aef5027ff740cd06fdaa4a1f
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041598,1305617503320.eecfd47c62f16f324e734e831b039cfa.: disabling compactions & flushes
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041398,1305617501838.7a717d98aef5027ff740cd06fdaa4a1f.: disabling compactions & flushes
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041528,1305617502805.8ed760e2d9c762a6bda7e9f99f2cd340.: disabling compactions & flushes
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041528,1305617502805.8ed760e2d9c762a6bda7e9f99f2cd340.
2011-05-25 09:15:00,777 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041622,1305617503507.bd07db88de68c1022c5bba9e5471c9f9.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,779 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=8feeb452cb116c37fcc8979ebf247d8c
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,777 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=d82ae0fd78acab7f21a3a92feef9259a
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,779 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=0f231435a5030975e8e4daca72b077b7
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041560,1305617503039.8feeb452cb116c37fcc8979ebf247d8c.: disabling compactions & flushes
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041560,1305617503039.8feeb452cb116c37fcc8979ebf247d8c.
2011-05-25 09:15:00,779 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041528,1305617502805.8ed760e2d9c762a6bda7e9f99f2cd340.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041464,1305617502337.0f231435a5030975e8e4daca72b077b7.: disabling compactions & flushes
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041398,1305617501838.7a717d98aef5027ff740cd06fdaa4a1f.
2011-05-25 09:15:00,779 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=b602cf813341bc877c354f10b6b60699
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=3b6332d456834e87c40d8e6f5300413e
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041598,1305617503320.eecfd47c62f16f324e734e831b039cfa.
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041348,1305617501463.3b6332d456834e87c40d8e6f5300413e.: disabling compactions & flushes
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041348,1305617501463.3b6332d456834e87c40d8e6f5300413e.
2011-05-25 09:15:00,779 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=eead635e232bd82c0322d0fb0df8cc81
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041150,1305617499950.eead635e232bd82c0322d0fb0df8cc81.: disabling compactions & flushes
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=13f0d76a7dc41ac90356c44eeb544848
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041228,1305617500543.7b510ae3fc4047e3b33d33b9d1101fbc.: disabling compactions & flushes
2011-05-25 09:15:00,778 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=956131b73ee2a892252607d860e2890f
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041090,1305617499498.956131b73ee2a892252607d860e2890f.: disabling compactions & flushes
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=2336491e5570f0d4ac6ddc41d1e2e62d
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=6822bdfb18f1dcb76d9a3bcbde50d99a
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=8123e5b472516d5f3773f663bb7d7b5f
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=e722cd712a2e97e370102c566f6a2589
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,778 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041412,1305617501947.044a911c7b6cc2c2e8583c7eda97f87e.: disabling compactions & flushes
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041202,1305617500340.e722cd712a2e97e370102c566f6a2589.: disabling compactions & flushes
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=6af74e3e07046b7865f2798ac078af24
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041390,1305617501775.6af74e3e07046b7865f2798ac078af24.: disabling compactions & flushes
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041524,1305617502774.8123e5b472516d5f3773f663bb7d7b5f.: disabling compactions & flushes
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041524,1305617502774.8123e5b472516d5f3773f663bb7d7b5f.
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=6c01970b64e79b0d55d81df36668b64e
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,782 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=9f668a0af4c07b93411ffda613f54462
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041332,1305617501338.6822bdfb18f1dcb76d9a3bcbde50d99a.: disabling compactions & flushes
2011-05-25 09:15:00,782 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=b9b089e842c1819423b82bd607f90fbb
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,782 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=8518b8df16f9ea956310b577cbe19e4b
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041436,1305617502134.2336491e5570f0d4ac6ddc41d1e2e62d.: disabling compactions & flushes
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041090,1305617499498.956131b73ee2a892252607d860e2890f.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041436,1305617502134.2336491e5570f0d4ac6ddc41d1e2e62d.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,782 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041090,1305617499498.956131b73ee2a892252607d860e2890f.
2011-05-25 09:15:00,782 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041436,1305617502134.2336491e5570f0d4ac6ddc41d1e2e62d.
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041370,1305617501619.13f0d76a7dc41ac90356c44eeb544848.: disabling compactions & flushes
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041370,1305617501619.13f0d76a7dc41ac90356c44eeb544848.
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=e3bfb1f005d4cef0245cd660ca4e75e7
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041228,1305617500543.7b510ae3fc4047e3b33d33b9d1101fbc.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=2903e26965a5b38bc4c7143c34718689
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041150,1305617499950.eead635e232bd82c0322d0fb0df8cc81.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041122,1305617499747.2903e26965a5b38bc4c7143c34718689.: disabling compactions & flushes
2011-05-25 09:15:00,780 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041422,1305617502025.e77f1c70062156eb4dbba21e3a65ee07.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041666,1305617503866.02fba8bdc638ea802ff3fb6a20c86445.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 02fba8bdc638ea802ff3fb6a20c86445 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=2c805b47fc2589a12fe8cac4cf6ea20a
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041372,1305617501635.2c805b47fc2589a12fe8cac4cf6ea20a.: disabling compactions & flushes
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041372,1305617501635.2c805b47fc2589a12fe8cac4cf6ea20a.
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041348,1305617501463.3b6332d456834e87c40d8e6f5300413e.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041690,1305617504037.a525cf0114894ee4663a9ada610f65e3.
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,780 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=9cb5ae4e48c1b3bbf1662781b843ea72
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,780 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041214,1305617500434.b602cf813341bc877c354f10b6b60699.: disabling compactions & flushes
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041214,1305617500434.b602cf813341bc877c354f10b6b60699.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041300,1305617501089.9cb5ae4e48c1b3bbf1662781b843ea72.: disabling compactions & flushes
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041300,1305617501089.9cb5ae4e48c1b3bbf1662781b843ea72.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041398,1305617501838.7a717d98aef5027ff740cd06fdaa4a1f.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041464,1305617502337.0f231435a5030975e8e4daca72b077b7.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041640,1305617503632.211c7387e7d36db3b463d8c31df34fd4.
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 211c7387e7d36db3b463d8c31df34fd4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,779 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041344,1305617501432.d82ae0fd78acab7f21a3a92feef9259a.: disabling compactions & flushes
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041344,1305617501432.d82ae0fd78acab7f21a3a92feef9259a.
2011-05-25 09:15:00,779 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=95a0e5b1bb0a2bbfa1f42ca21fb27656
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041572,1305617503132.95a0e5b1bb0a2bbfa1f42ca21fb27656.: disabling compactions & flushes
2011-05-25 09:15:00,784 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041560,1305617503039.8feeb452cb116c37fcc8979ebf247d8c.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041708,1305617504162.842998665aed082207cda149fa6af4a6.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 842998665aed082207cda149fa6af4a6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041694,1305617504068.d2546cde1d6546133af74f25252b652c.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041300,1305617501089.9cb5ae4e48c1b3bbf1662781b843ea72.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041710,1305617504178.11411507d349f663d4a0d3179adc348b.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,784 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041598,1305617503320.eecfd47c62f16f324e734e831b039cfa.
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node a525cf0114894ee4663a9ada610f65e3 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,784 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041372,1305617501635.2c805b47fc2589a12fe8cac4cf6ea20a.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041122,1305617499747.2903e26965a5b38bc4c7143c34718689.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041122,1305617499747.2903e26965a5b38bc4c7143c34718689.
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041150,1305617499950.eead635e232bd82c0322d0fb0df8cc81.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041800,1305617504880.3c77a6fd6c1595b4275432e51e710c0f.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 3c77a6fd6c1595b4275432e51e710c0f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,783 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041492,1305617502540.e3bfb1f005d4cef0245cd660ca4e75e7.: disabling compactions & flushes
2011-05-25 09:15:00,783 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041228,1305617500543.7b510ae3fc4047e3b33d33b9d1101fbc.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041816,1305617505004.e1e930bda75800eb54f7ecaf508dcfd8.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node e1e930bda75800eb54f7ecaf508dcfd8 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041370,1305617501619.13f0d76a7dc41ac90356c44eeb544848.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041818,1305617505020.2a40e387d930cf6a18e2ab2312196936.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041646,1305617503694.baa7bc348497dbffc56e9cfd5d1d5f00.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041642,1305617503647.e307790d5205ec2b9bff52c7bc82b05a.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node baa7bc348497dbffc56e9cfd5d1d5f00 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node e307790d5205ec2b9bff52c7bc82b05a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041032,1305617499076.8518b8df16f9ea956310b577cbe19e4b.: disabling compactions & flushes
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041032,1305617499076.8518b8df16f9ea956310b577cbe19e4b.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041032,1305617499076.8518b8df16f9ea956310b577cbe19e4b.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041824,1305617505067.02f657b6a3641c68798d911215ac4c39.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 02f657b6a3641c68798d911215ac4c39 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041088,1305617499498.b9b089e842c1819423b82bd607f90fbb.: disabling compactions & flushes
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041088,1305617499498.b9b089e842c1819423b82bd607f90fbb.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041088,1305617499498.b9b089e842c1819423b82bd607f90fbb.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041844,1305617505207.8fa1872af7f997a676440fca4187bf9a.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 8fa1872af7f997a676440fca4187bf9a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041062,1305617499295.9f668a0af4c07b93411ffda613f54462.: disabling compactions & flushes
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041062,1305617499295.9f668a0af4c07b93411ffda613f54462.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041332,1305617501338.6822bdfb18f1dcb76d9a3bcbde50d99a.
2011-05-25 09:15:00,782 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041050,1305617499201.6c01970b64e79b0d55d81df36668b64e.: disabling compactions & flushes
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041390,1305617501775.6af74e3e07046b7865f2798ac078af24.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041390,1305617501775.6af74e3e07046b7865f2798ac078af24.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041848,1305617505238.bd6f7abb7664363cc1a45949015d809b.
2011-05-25 09:15:00,781 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=dfbb69117ef503bfabe2761314dfd956
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:518)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:419)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:272)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:331)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:364)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1354)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:215)
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041202,1305617500340.e722cd712a2e97e370102c566f6a2589.
2011-05-25 09:15:00,781 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041412,1305617501947.044a911c7b6cc2c2e8583c7eda97f87e.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing ufdr,041074,1305617499388.dfbb69117ef503bfabe2761314dfd956.: disabling compactions & flushes
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd6f7abb7664363cc1a45949015d809b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041074,1305617499388.dfbb69117ef503bfabe2761314dfd956.
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041524,1305617502774.8123e5b472516d5f3773f663bb7d7b5f.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041050,1305617499201.6c01970b64e79b0d55d81df36668b64e.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041050,1305617499201.6c01970b64e79b0d55d81df36668b64e.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041332,1305617501338.6822bdfb18f1dcb76d9a3bcbde50d99a.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 2a40e387d930cf6a18e2ab2312196936 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region ufdr,041492,1305617502540.e3bfb1f005d4cef0245cd660ca4e75e7.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041492,1305617502540.e3bfb1f005d4cef0245cd660ca4e75e7.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041782,1305617504739.ff13e1ec81741400540b185aa4ec780b.
2011-05-25 09:15:00,785 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,041754,1305617504521.c274d87c45fde52ed138a39661b1760f.
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c274d87c45fde52ed138a39661b1760f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d4a9b331665d994f9c756d1d9bcc197d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,787 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041074,1305617499388.dfbb69117ef503bfabe2761314dfd956.
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,042086,1305617507064.b2d218a727ca9cbbc9cb7959f6747d02.
2011-05-25 09:15:00,786 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041202,1305617500340.e722cd712a2e97e370102c566f6a2589.
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,042092,1305617507111.4c2735058af5eefef41adb345ef5b50b.
2011-05-25 09:15:00,786 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed value
2011-05-25 09:15:00,788 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 4c2735058af5eefef41adb345ef5b50b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node b2d218a727ca9cbbc9cb7959f6747d02 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node e9bbae60f4bd7373a1af2d68f1aa9bdc from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,787 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041572,1305617503132.95a0e5b1bb0a2bbfa1f42ca21fb27656.
2011-05-25 09:15:00,788 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed ufdr,041412,1305617501947.044a911c7b6cc2c2e8583c7eda97f87e.
2011-05-25 09:15:00,788 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,042094,1305617507126.a09a0661045aa1002ed9c6d70e73e387.
2011-05-25 09:15:00,788 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of ufdr,042098,1305617507142.36d79ac3a52dfa036cf8639d3cb81800.
2011-05-25 09:15:00,788 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node a09a0661045aa1002ed9c6d70e73e387 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,788 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 36d79ac3a52dfa036cf8639d3cb81800 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,796 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,796 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041624,1305617503522.c4c657deda864fa2018b38388ee928e6.', STARTKEY => '041624', ENDKEY => '041626', ENCODED => c4c657deda864fa2018b38388ee928e6, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,797 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041624,1305617503522.c4c657deda864fa2018b38388ee928e6.
2011-05-25 09:15:00,802 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr,041624,1305617503522.c4c657deda864fa2018b38388ee928e6.; next sequenceid=1
2011-05-25 09:15:00,802 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,804 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node bd07db88de68c1022c5bba9e5471c9f9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,805 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041622,1305617503507.bd07db88de68c1022c5bba9e5471c9f9.', STARTKEY => '041622', ENDKEY => '041624', ENCODED => bd07db88de68c1022c5bba9e5471c9f9, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,805 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041622,1305617503507.bd07db88de68c1022c5bba9e5471c9f9.
2011-05-25 09:15:00,808 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr,041622,1305617503507.bd07db88de68c1022c5bba9e5471c9f9.; next sequenceid=1
2011-05-25 09:15:00,808 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,835 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 02fba8bdc638ea802ff3fb6a20c86445 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,835 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 211c7387e7d36db3b463d8c31df34fd4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,835 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 842998665aed082207cda149fa6af4a6 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,835 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041666,1305617503866.02fba8bdc638ea802ff3fb6a20c86445.', STARTKEY => '041666', ENDKEY => '041668', ENCODED => 02fba8bdc638ea802ff3fb6a20c86445, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,835 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041640,1305617503632.211c7387e7d36db3b463d8c31df34fd4.', STARTKEY => '041640', ENDKEY => '041642', ENCODED => 211c7387e7d36db3b463d8c31df34fd4, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041708,1305617504162.842998665aed082207cda149fa6af4a6.', STARTKEY => '041708', ENDKEY => '041710', ENCODED => 842998665aed082207cda149fa6af4a6, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 3c77a6fd6c1595b4275432e51e710c0f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041800,1305617504880.3c77a6fd6c1595b4275432e51e710c0f.', STARTKEY => '041800', ENDKEY => '041802', ENCODED => 3c77a6fd6c1595b4275432e51e710c0f, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node a525cf0114894ee4663a9ada610f65e3 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041666,1305617503866.02fba8bdc638ea802ff3fb6a20c86445.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041694,1305617504068.d2546cde1d6546133af74f25252b652c.', STARTKEY => '041694', ENDKEY => '041696', ENCODED => d2546cde1d6546133af74f25252b652c, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node e1e930bda75800eb54f7ecaf508dcfd8 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041708,1305617504162.842998665aed082207cda149fa6af4a6.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041816,1305617505004.e1e930bda75800eb54f7ecaf508dcfd8.', STARTKEY => '041816', ENDKEY => '041818', ENCODED => e1e930bda75800eb54f7ecaf508dcfd8, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node baa7bc348497dbffc56e9cfd5d1d5f00 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041690,1305617504037.a525cf0114894ee4663a9ada610f65e3.', STARTKEY => '041690', ENDKEY => '041692', ENCODED => a525cf0114894ee4663a9ada610f65e3, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041646,1305617503694.baa7bc348497dbffc56e9cfd5d1d5f00.', STARTKEY => '041646', ENDKEY => '041648', ENCODED => baa7bc348497dbffc56e9cfd5d1d5f00, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node e307790d5205ec2b9bff52c7bc82b05a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041694,1305617504068.d2546cde1d6546133af74f25252b652c.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 8fa1872af7f997a676440fca4187bf9a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041844,1305617505207.8fa1872af7f997a676440fca4187bf9a.', STARTKEY => '041844', ENDKEY => '041846', ENCODED => 8fa1872af7f997a676440fca4187bf9a, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041690,1305617504037.a525cf0114894ee4663a9ada610f65e3.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 2a40e387d930cf6a18e2ab2312196936 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041800,1305617504880.3c77a6fd6c1595b4275432e51e710c0f.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041640,1305617503632.211c7387e7d36db3b463d8c31df34fd4.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041818,1305617505020.2a40e387d930cf6a18e2ab2312196936.', STARTKEY => '041818', ENDKEY => '041820', ENCODED => 2a40e387d930cf6a18e2ab2312196936, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c274d87c45fde52ed138a39661b1760f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11411507d349f663d4a0d3179adc348b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041844,1305617505207.8fa1872af7f997a676440fca4187bf9a.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041754,1305617504521.c274d87c45fde52ed138a39661b1760f.', STARTKEY => '041754', ENDKEY => '041756', ENCODED => c274d87c45fde52ed138a39661b1760f, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 203e1dabddae5da2327b46cdf9595b25 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041818,1305617505020.2a40e387d930cf6a18e2ab2312196936.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 733bf8935e622a2625aaec5a72700310 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node ff13e1ec81741400540b185aa4ec780b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041816,1305617505004.e1e930bda75800eb54f7ecaf508dcfd8.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041894,1305617505582.733bf8935e622a2625aaec5a72700310.', STARTKEY => '041894', ENDKEY => '041896', ENCODED => 733bf8935e622a2625aaec5a72700310, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041782,1305617504739.ff13e1ec81741400540b185aa4ec780b.', STARTKEY => '041782', ENDKEY => '041784', ENCODED => ff13e1ec81741400540b185aa4ec780b, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041646,1305617503694.baa7bc348497dbffc56e9cfd5d1d5f00.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node bd6f7abb7664363cc1a45949015d809b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041642,1305617503647.e307790d5205ec2b9bff52c7bc82b05a.', STARTKEY => '041642', ENDKEY => '041644', ENCODED => e307790d5205ec2b9bff52c7bc82b05a, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 32226f3e1cb7b9c99ecf109154887b7e from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041848,1305617505238.bd6f7abb7664363cc1a45949015d809b.', STARTKEY => '041848', ENDKEY => '041850', ENCODED => bd6f7abb7664363cc1a45949015d809b, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 4bf9274e2d81a054e7c3249bf9d91d5c from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041876,1305617505457.32226f3e1cb7b9c99ecf109154887b7e.', STARTKEY => '041876', ENDKEY => '041878', ENCODED => 32226f3e1cb7b9c99ecf109154887b7e, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041894,1305617505582.733bf8935e622a2625aaec5a72700310.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042048,1305617506783.4bf9274e2d81a054e7c3249bf9d91d5c.', STARTKEY => '042048', ENDKEY => '042050', ENCODED => 4bf9274e2d81a054e7c3249bf9d91d5c, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 4c2735058af5eefef41adb345ef5b50b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node b2d218a727ca9cbbc9cb7959f6747d02 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042092,1305617507111.4c2735058af5eefef41adb345ef5b50b.', STARTKEY => '042092', ENDKEY => '042094', ENCODED => 4c2735058af5eefef41adb345ef5b50b, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042086,1305617507064.b2d218a727ca9cbbc9cb7959f6747d02.', STARTKEY => '042086', ENDKEY => '042088', ENCODED => b2d218a727ca9cbbc9cb7959f6747d02, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041848,1305617505238.bd6f7abb7664363cc1a45949015d809b.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d4a9b331665d994f9c756d1d9bcc197d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 02f657b6a3641c68798d911215ac4c39 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041872,1305617505426.d4a9b331665d994f9c756d1d9bcc197d.', STARTKEY => '041872', ENDKEY => '041874', ENCODED => d4a9b331665d994f9c756d1d9bcc197d, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041824,1305617505067.02f657b6a3641c68798d911215ac4c39.', STARTKEY => '041824', ENDKEY => '041826', ENCODED => 02f657b6a3641c68798d911215ac4c39, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 36d79ac3a52dfa036cf8639d3cb81800 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042092,1305617507111.4c2735058af5eefef41adb345ef5b50b.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042098,1305617507142.36d79ac3a52dfa036cf8639d3cb81800.', STARTKEY => '042098', ENDKEY => '042100', ENCODED => 36d79ac3a52dfa036cf8639d3cb81800, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042086,1305617507064.b2d218a727ca9cbbc9cb7959f6747d02.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042048,1305617506783.4bf9274e2d81a054e7c3249bf9d91d5c.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041876,1305617505457.32226f3e1cb7b9c99ecf109154887b7e.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node e9bbae60f4bd7373a1af2d68f1aa9bdc from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041642,1305617503647.e307790d5205ec2b9bff52c7bc82b05a.
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041870,1305617505410.e9bbae60f4bd7373a1af2d68f1aa9bdc.', STARTKEY => '041870', ENDKEY => '041872', ENCODED => e9bbae60f4bd7373a1af2d68f1aa9bdc, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041782,1305617504739.ff13e1ec81741400540b185aa4ec780b.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 26c9e468cb0684b3e953f646df858dab from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041754,1305617504521.c274d87c45fde52ed138a39661b1760f.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041930,1305617505863.203e1dabddae5da2327b46cdf9595b25.', STARTKEY => '041930', ENDKEY => '041932', ENCODED => 203e1dabddae5da2327b46cdf9595b25, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041870,1305617505410.e9bbae60f4bd7373a1af2d68f1aa9bdc.
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 34fd577551f324fc601f2a11fe310994 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 4772be23e4f86e295bc3d25d19aa81b2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042004,1305617506455.34fd577551f324fc601f2a11fe310994.', STARTKEY => '042004', ENDKEY => '042006', ENCODED => 34fd577551f324fc601f2a11fe310994, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,836 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041710,1305617504178.11411507d349f663d4a0d3179adc348b.', STARTKEY => '041710', ENDKEY => '041712', ENCODED => 11411507d349f663d4a0d3179adc348b, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041726,1305617504302.4772be23e4f86e295bc3d25d19aa81b2.', STARTKEY => '041726', ENDKEY => '041728', ENCODED => 4772be23e4f86e295bc3d25d19aa81b2, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041930,1305617505863.203e1dabddae5da2327b46cdf9595b25.
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,041888,1305617505551.26c9e468cb0684b3e953f646df858dab.', STARTKEY => '041888', ENDKEY => '041890', ENCODED => 26c9e468cb0684b3e953f646df858dab, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042098,1305617507142.36d79ac3a52dfa036cf8639d3cb81800.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041824,1305617505067.02f657b6a3641c68798d911215ac4c39.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041872,1305617505426.d4a9b331665d994f9c756d1d9bcc197d.
2011-05-25 09:15:00,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node a09a0661045aa1002ed9c6d70e73e387 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041726,1305617504302.4772be23e4f86e295bc3d25d19aa81b2.
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'ufdr,042094,1305617507126.a09a0661045aa1002ed9c6d70e73e387.', STARTKEY => '042094', ENDKEY => '042096', ENCODED => a09a0661045aa1002ed9c6d70e73e387, TABLE => {{NAME => 'ufdr', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2592000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041888,1305617505551.26c9e468cb0684b3e953f646df858dab.
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,041710,1305617504178.11411507d349f663d4a0d3179adc348b.
2011-05-25 09:15:00,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042004,1305617506455.34fd577551f324fc601f2a11fe310994.
2011-05-25 09:15:00,839 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated ufdr,042094,1305617507126.a09a0661045aa1002ed9c6d70e73e387.
2011-05-25 09:15:00,845 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr,041694,1305617504068.d2546cde1d6546133af74f25252b652c.; next sequenceid=1
2011-05-25 09:15:00,845 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,846 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr,042092,1305617507111.4c2735058af5eefef41adb345ef5b50b.; next sequenceid=1
2011-05-25 09:15:00,853 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node b2d218a727ca9cbbc9cb7959f6747d02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,858 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,871 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,891 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr,042004,1305617506455.34fd577551f324fc601f2a11fe310994.; next sequenceid=1
2011-05-25 09:15:00,891 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 34fd577551f324fc601f2a11fe310994 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:00,897 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 4c2735058af5eefef41adb345ef5b50b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:04,255 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 733bf8935e622a2625aaec5a72700310 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:06,314 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:06,329 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:06,415 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,261908,1305617924238.6e797c03faea7d1a0cb4973661ae1fed.
2011-05-25 09:15:06,663 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,052226,1305617526845.0ef0b63ff689678b282ca9f1b9dd5b05.
2011-05-25 09:15:07,517 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:07,525 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:07,533 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:07,538 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:07,544 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:07,553 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:14,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node b2d218a727ca9cbbc9cb7959f6747d02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:14,425 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 733bf8935e622a2625aaec5a72700310 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:14,843 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,270344,1305617930797.a235377bbf86ea38cba721f6b853bd67.
2011-05-25 09:15:14,843 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,080744,1305617575521.54612f5fe5058cfc45f98386a6346800.
2011-05-25 09:15:14,844 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,071018,1305617557203.b4410b1759febe09007392c6ee30150e.
2011-05-25 09:15:15,180 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,080772,1305617575724.9efe8eb60c238e7be1982974e597ac10.
2011-05-25 09:15:15,182 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,270084,1305617928800.18cbe77e89a153c5627001baaf5a5f53.
2011-05-25 09:15:16,350 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:16,361 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:17,558 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:17,566 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:17,573 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:17,575 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:24,513 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 34fd577551f324fc601f2a11fe310994 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:24,513 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node e1e930bda75800eb54f7ecaf508dcfd8 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:24,522 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node b2d218a727ca9cbbc9cb7959f6747d02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:24,522 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 733bf8935e622a2625aaec5a72700310 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:26,160 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,271922,1305617942497.72815a19beef809b8a0fb79be0a24194.
2011-05-25 09:15:26,380 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:26,392 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,293 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 733bf8935e622a2625aaec5a72700310 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,293 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node b2d218a727ca9cbbc9cb7959f6747d02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,314 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node e1e930bda75800eb54f7ecaf508dcfd8 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,314 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node ff13e1ec81741400540b185aa4ec780b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,314 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 2a40e387d930cf6a18e2ab2312196936 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,315 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 34fd577551f324fc601f2a11fe310994 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,328 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node b2d218a727ca9cbbc9cb7959f6747d02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:41,328 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 733bf8935e622a2625aaec5a72700310 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Thanks!
Jieshan Bean
-----------------------------------------------
Can you post the full log somewhere? You talk about several Exceptions
but we can't see them.
J-D
On Tue, May 31, 2011 at 4:41 AM, bijieshan <bi...@huawei.com> wrote:
> It occurred in an RegionServer with an un-known reason. I have check this RegionServer logs, there's no prev-aborting, and no other infos showed the RegionServer has aborted. So I saw the following msg showed in a sudden.
>
>>>[logs]
> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>>>[description]
> Before that time, everything seemed ok(it received so many open-region requests, and handled them), except several Exception happened during the time.
>
>>>[logs]
> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,110018,1305617626846.f22eafaf06c3be7f7e5a447cec40c7fa.
> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,111336,1305617636659.b47f1cf09b10aecaee6805a985646b30.
> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280884,1305617952747.7cac8ccc9c9770ef0e48807e07bc8bec.
> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280446,1305617949439.b332cfff565d77b2e2d4f1a2c52494b0.
> 2011-05-25 09:15:36,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,101974,1305617623149.b60c71565f2c73b81477fe25ba80378b.
> 2011-05-25 09:15:36,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:36,423 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,660 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,662 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>
>>>[description]
> From the code, I saw that , only the Server has aborted, the msg of "Server not running, aborting" could be shown. It determined by the param of HRegionServer#abortRequested, if it was true, and if somewhere called the method of HRegionServer#checkOpen, this msg would been printed.
>
> /**
> * Called to verify that this server is up and running.
> *
> * @throws IOException
> */
> protected void checkOpen() throws IOException {
> if (this.stopped || this.abortRequested) {
> throw new IOException("Server not running"
> + (this.abortRequested ? ", aborting" : ""));
> }
> if (!fsOk) {
> throw new IOException("File system not available");
> }
> }
>
> For I said there's no aborted info been found in Logs:
>
> public void abort(String reason, Throwable cause) {
> if (cause != null) {
> LOG.fatal("ABORTING region server " + this + ": " + reason, cause);
> } else {
> LOG.fatal("ABORTING region server " + this + ": " + reason);
> }
> this.abortRequested = true;
> this.reservedSpace.clear();
> if (this.metrics != null) {
> LOG.info("Dump of metrics: " + this.metrics);
> }
> stop(reason);
> }
>
> For there's no prev-aborting found in the logs, how did the sudden msg of " java.io.IOException: Server not running, aborting " occurred?
> Thanks!
>
> Jieshan Bean.
>
>
>
>
>
Re: A sudden msg of "java.io.IOException: Server not running, aborting"
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Can you post the full log somewhere? You talk about several Exceptions
but we can't see them.
J-D
On Tue, May 31, 2011 at 4:41 AM, bijieshan <bi...@huawei.com> wrote:
> It occurred in an RegionServer with an un-known reason. I have check this RegionServer logs, there's no prev-aborting, and no other infos showed the RegionServer has aborted. So I saw the following msg showed in a sudden.
>
>>>[logs]
> 2011-05-25 09:15:44,232 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Running rollback of failed split of ufdr,0065286138106876#4228000,1306260358978.37875b35a870957da534ad29fd2944d5.; java.io.IOException: Server not running, aborting
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2352)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1653)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>>>[description]
> Before that time, everything seemed ok(it received so many open-region requests, and handled them), except several Exception happened during the time.
>
>>>[logs]
> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,110018,1305617626846.f22eafaf06c3be7f7e5a447cec40c7fa.
> 2011-05-25 09:15:36,214 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,111336,1305617636659.b47f1cf09b10aecaee6805a985646b30.
> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280884,1305617952747.7cac8ccc9c9770ef0e48807e07bc8bec.
> 2011-05-25 09:15:36,215 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,280446,1305617949439.b332cfff565d77b2e2d4f1a2c52494b0.
> 2011-05-25 09:15:36,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: ufdr,101974,1305617623149.b60c71565f2c73b81477fe25ba80378b.
> 2011-05-25 09:15:36,413 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:36,423 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node 11dc72d94c7a5a3d19b0c0c3c49624a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,660 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node c4c657deda864fa2018b38388ee928e6 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,662 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Attempting to transition node bd07db88de68c1022c5bba9e5471c9f9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-05-25 09:15:37,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:20020-0x32ffbc7e6e00321 Successfully transitioned node d2546cde1d6546133af74f25252b652c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
>
>>>[description]
> From the code, I saw that , only the Server has aborted, the msg of "Server not running, aborting" could be shown. It determined by the param of HRegionServer#abortRequested, if it was true, and if somewhere called the method of HRegionServer#checkOpen, this msg would been printed.
>
> /**
> * Called to verify that this server is up and running.
> *
> * @throws IOException
> */
> protected void checkOpen() throws IOException {
> if (this.stopped || this.abortRequested) {
> throw new IOException("Server not running"
> + (this.abortRequested ? ", aborting" : ""));
> }
> if (!fsOk) {
> throw new IOException("File system not available");
> }
> }
>
> For I said there's no aborted info been found in Logs:
>
> public void abort(String reason, Throwable cause) {
> if (cause != null) {
> LOG.fatal("ABORTING region server " + this + ": " + reason, cause);
> } else {
> LOG.fatal("ABORTING region server " + this + ": " + reason);
> }
> this.abortRequested = true;
> this.reservedSpace.clear();
> if (this.metrics != null) {
> LOG.info("Dump of metrics: " + this.metrics);
> }
> stop(reason);
> }
>
> For there's no prev-aborting found in the logs, how did the sudden msg of " java.io.IOException: Server not running, aborting " occurred?
> Thanks!
>
> Jieshan Bean.
>
>
>
>
>