You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Yossi Ittach (JIRA)" <ji...@apache.org> on 2008/12/29 17:20:44 UTC
[jira] Created: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
---------------------------------------------------------------------------------------------------------------
Key: HBASE-1094
URL: https://issues.apache.org/jira/browse/HBASE-1094
Project: Hadoop HBase
Issue Type: Bug
Components: client
Affects Versions: 0.18.1, 0.18.0
Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
Reporter: Yossi Ittach
After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
A restart of the client usually fixes the problem , but it happens again after a while.
LogFiles excerpts:
*RegionServer*
2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
*Master:*
2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount of
file inserts)
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Guilherme,
As Stack suggested, please do so.
J-D
On Tue, Mar 10, 2009 at 7:55 AM, Guilherme Germoglio
<ge...@gmail.com> wrote:
> I'm sorry the newbie question but should I create this new issue?
>
> I'm just a novice tinkering around HBase looking for an opportunity to
> contribute to such a great project (and, yes, I'm also looking for a project
> for Google Summer of Code).
>
> Thank you
>
> On Sun, Mar 8, 2009 at 6:17 PM, Guilherme Mauro Germoglio Barbosa (JIRA) <
> jira@apache.org> wrote:
>
>>
>> Â Â [
>> https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680016#action_12680016]
>>
>> Guilherme Mauro Germoglio Barbosa commented on HBASE-1094:
>> ----------------------------------------------------------
>>
>> I think so.
>>
>> Also, I've edited HConnectionManager in order to use all HBaseConfiguration
>> instead of only HBASE_DIR as key on the HBASE_INSTANCES map. To do so, I've
>> implemented HBaseConfiguration.hashcode() -- just like a HashMap.hashCode()
>> is implemented -- and all tests are green with these two modifications. So,
>> I can upload the patch as soon as the issue is created.
>>
>> > Region Server throw NotServingRegionException when batchUpdate irredicly
>> (After a large amount of file inserts)
>> >
>> ---------------------------------------------------------------------------------------------------------------
>> >
>> > Â Â Â Â Â Â Â Â Key: HBASE-1094
>> > Â Â Â Â Â Â Â Â URL: https://issues.apache.org/jira/browse/HBASE-1094
>> > Â Â Â Â Â Â Project: Hadoop HBase
>> > Â Â Â Â Â Issue Type: Bug
>> > Â Â Â Â Â Components: client
>> > Â Â Affects Versions: 0.18.0, 0.18.1
>> > Â Â Â Â Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2
>> RegionServers
>> > Â Â Â Â Â Â Reporter: Yossi Ittach
>> > Â Â Â Â Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG,
>> hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log,
>> RSbugLog.log
>> >
>> >
>> > After a large amount of file (~400,000) , once in a while during rapid
>> inserts RegionServer returns NotServingRegionException when doing
>> batchUpdate.
>> > A restart of the client usually fixes the problem , but it happens again
>> after a while.
>> > LogFiles excerpts:
>> > *RegionServer*
>> > 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 1333
>> > 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 1334
>> > 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333
>> size:65557 offsetInBlock:19507200 lastPacketInBlock:false
>> > 2008-12-16 09:17:14,439 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
>> > 2008-12-16 09:17:14,441 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,461 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
>> disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,461 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,461 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
>> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,461 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,462 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
>> on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,462 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for
>> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region
>> memcache size 236.3k
>> > 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 0
>> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 1
>> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.211:50010
>> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0
>> size:65557 offsetInBlock:0 lastPacketInBlock:false
>> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 1
>> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 2
>> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1
>> size:65557 offsetInBlock:65024 lastPacketInBlock:false
>> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 2
>> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 3
>> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2
>> size:65557 offsetInBlock:130048 lastPacketInBlock:false
>> > 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3
>> size:49273 offsetInBlock:195072 lastPacketInBlock:true
>> > 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 1
>> > 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 2
>> > 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 3
>> > 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_4611894842462358439_1544
>> > 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.211:50010
>> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0
>> size:276 offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_-6152926948595840609_1544
>> > 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/obde_content/info/5474929915463365960:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38
>> offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_2767846628464404081_1545
>> > 2008-12-16 09:17:14,644 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: Added
>> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56
>> entries, sequence id 235152, data size 236.3k, file size 238.2k
>> > 2008-12-16 09:17:14,644 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for
>> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence
>> id=235152, compaction requested=true
>> > 2008-12-16 09:17:14,644 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
>> > 2008-12-16 09:17:14,644 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: closed
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 4 on 60020, call batchUpdate([B@552a2a7d, row =>
>> f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0
>> size:84 offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_-4579777298287321197_1547
>> > 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 9 on 60020, call batchUpdate([B@18084038, row =>
>> hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 8 on 60020, call batchUpdate([B@74da3b58, row =>
>> rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 60020, call batchUpdate([B@51575d48, row =>
>> aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 60020, call batchUpdate([B@63442ff5, row =>
>> iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 6 on 60020, call batchUpdate([B@59eb5159, row =>
>> t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 60020, call batchUpdate([B@132fa7c8, row =>
>> Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 1333
>> > 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 60020, call batchUpdate([B@4696de68, row =>
>> Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 7 on 60020, call batchUpdate([B@16a3f072, row =>
>> gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 1334
>> > 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334
>> size:65557 offsetInBlock:19572224 lastPacketInBlock:false
>> > 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 9 on 60020, call batchUpdate([B@171591e3, row =>
>> ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 8 on 60020, call batchUpdate([B@452719a0, row =>
>> bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 2 on 60020, call batchUpdate([B@6a76000a, row =>
>> XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 1 on 60020, call batchUpdate([B@2b753bb9, row =>
>> jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 60020, call batchUpdate([B@7b4286a2, row =>
>> VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 1335
>> > 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0
>> size:84 offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_-1532247345865982631_1549
>> > 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 60020, call batchUpdate([B@3f229bc1, row =>
>> WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84
>> offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_9044590686588152812_1551
>> > 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 8 on 60020, call batchUpdate([B@74d3776e, row =>
>> hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0
>> size:84 offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_-3568743239673676944_1553
>> > 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 3 on 60020, call batchUpdate([B@366480d7, row =>
>> j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 7 on 60020, call batchUpdate([B@24af9995, row =>
>> z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0
>> size:84 offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_-1561317920938610537_1555
>> > 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 60020, call batchUpdate([B@46823b18, row =>
>> uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 0 on 60020, call batchUpdate([B@4718f92, row =>
>> VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > org.apache.hadoop.hbase.NotServingRegionException: Region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient:
>> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010:
>> masked=rwxr-xr-x
>> > 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 0
>> > 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
>> new block
>> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.102:50010
>> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
>> MY.IP.213:50010
>> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
>> to MY.IP.102:50010
>> > 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
>> size 131071
>> > 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84
>> offsetInBlock:0 lastPacketInBlock:true
>> > 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> received ack for seqno 0
>> > 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
>> old block blk_5499006216568952565_1557
>> > 2008-12-16 09:17:16,081 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
>> > 2008-12-16 09:17:16,117 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010,
>> isReference=true, sequence id=207793
>> > 2008-12-16 09:17:16,126 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010,
>> isReference=true, sequence id=235152
>> > 2008-12-16 09:17:16,134 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010,
>> isReference=true, sequence id=235051
>> > 2008-12-16 09:17:16,135 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore
>> 1010339458/obde_content, max sequence id 235152
>> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
>> brand-new decompressor
>> > 2008-12-16 09:17:16,157 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
>> > 2008-12-16 09:17:16,159 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
>> disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
>> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
>> on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
>> > 2008-12-16 09:17:16,159 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: closed
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> > 2008-12-16 09:17:16,159 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
>> > 2008-12-16 09:17:16,185 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010,
>> isReference=true, sequence id=207793
>> > 2008-12-16 09:17:16,193 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010,
>> isReference=true, sequence id=235152
>> > 2008-12-16 09:17:16,202 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: loaded
>> /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010,
>> isReference=true, sequence id=235051
>> > 2008-12-16 09:17:16,203 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore
>> 2015194550/obde_content, max sequence id 235152
>> > 2008-12-16 09:17:16,229 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
>> > 2008-12-16 09:17:16,231 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
>> disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
>> region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
>> on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,231 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
>> > 2008-12-16 09:17:16,231 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: closed
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
>> > 2008-12-16 09:17:16,245 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up
>> /hbase/my_table/1608250010/splits true
>> > 2008-12-16 09:17:16,247 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
>> table locations for row <> and tableName .META.: location server
>> MY.IP.102:60020, location region name .META.,,1
>> > 2008-12-16 09:17:16,249 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
>> updated, and report to master all successful. Old region=REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
>> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
>> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
>> 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false',
>> LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460,
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
>> > 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 7 on 60020, call batchUpdate([B@4d2637c2, row =>
>> ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 6 on 60020, call batchUpdate([B@816c920, row =>
>> bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
>> > Â Â Â at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
>> > Â Â Â at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > Â Â Â at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > Â Â Â at java.lang.reflect.Method.invoke(Method.java:585)
>> > Â Â Â at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> > Â Â Â at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> > 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk packet full seqno 1335
>> > 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient:
>> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335
>> size:65557 offsetInBlock:19637248 lastPacketInBlock:false
>> > 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
>> writeChunk allocating new packet 1336
>> > 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 8 on 60020, call batchUpdate([B@fee5806, row =>
>> o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1)
>> from MY.IP.212:46198: error:
>> org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > *Master:*
>> > 2008-12-16 09:15:31,712 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE =>
>> 1229433746854
>> > 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:15:46,728 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:16:01,743 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:16:16,760 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:16:31,775 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE =>
>> 1229433746854
>> > 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:16:46,791 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:17:01,807 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:17:16,823 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
>> Avg Load: 2.0
>> > 2008-12-16 09:17:16,902 INFO
>> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from
>> MY.IP.102:60020
>> > 2008-12-16 09:17:19,826 INFO
>> org.apache.hadoop.hbase.master.RegionManager: assigning region
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
>> > 2008-12-16 09:17:19,828 INFO
>> org.apache.hadoop.hbase.master.RegionManager: assigning region
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 INFO
>> org.apache.hadoop.hbase.master.ServerManager: Received
>> MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
>> from MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 INFO
>> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 INFO
>> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster:
>> Main processing loop: PendingOpenOperation from MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 INFO
>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
>> > 2008-12-16 09:17:22,833 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 1
>> > 2008-12-16 09:17:22,833 INFO
>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with
>> startcode 1229433746880 and server MY.IP.213:60020
>> > 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster:
>> Main processing loop: PendingOpenOperation from MY.IP.213:60020
>> > 2008-12-16 09:17:22,835 INFO
>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
>> > 2008-12-16 09:17:22,835 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 1
>> > 2008-12-16 09:17:22,835 INFO
>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with
>> startcode 1229433746880 and server MY.IP.213:60020
>> > 2008-12-16 09:17:31,845 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
>> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
>> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
>> 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE =>
>> 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
>> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
>> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
>> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
>> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.213:60020', STARTCODE => 1229433746880
>> > 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
>> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
>> 1229433746880
>> > 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:17:46,864 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:18:01,879 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:18:16,895 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:18:31,912 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
>> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
>> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
>> 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE =>
>> 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
>> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
>> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
>> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
>> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.213:60020', STARTCODE => 1229433746880
>> > 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
>> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
>> 1229433746880
>> > 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to
>> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
>> > 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because
>> daughter splits no longer hold references
>> > 2008-12-16 09:18:37,858 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://
>> be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
>> > 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:18:46,927 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:19:01,943 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:19:16,960 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:19:31,976 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
>> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
>> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
>> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
>> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.213:60020', STARTCODE => 1229433746880
>> > 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
>> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
>> 1229433746880
>> > 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:19:46,992 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:20:02,006 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:20:17,022 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:20:32,037 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020}
>> > 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
>> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
>> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
>> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
>> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
>> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
>> => 1229433746854
>> > 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
>> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
>> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
>> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
>> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
>> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
>> 'MY.IP.213:60020', STARTCODE => 1229433746880
>> > 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner REGION => {NAME =>
>> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
>> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
>> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
>> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
>> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
>> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
>> 1229433746880
>> > 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
>> startKey: <>, server: MY.IP.102:60020} complete
>> > 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> all meta regions scanned
>> > 2008-12-16 09:20:47,052 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:21:02,069 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:21:17,084 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>> > 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020}
>> > 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
>> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
>> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
>> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
>> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
>> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
>> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
>> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
>> > 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
>> startKey: <>, server: MY.IP.213:60020} complete
>> > 2008-12-16 09:21:32,100 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
>> Avg Load: 3.0
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>
>
> --
> Guilherme
>
> msn: guigermoglio@hotmail.com
> homepage: http://germoglio.googlepages.com
>
Re: [jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount of
file inserts)
Posted by Guilherme Germoglio <ge...@gmail.com>.
I'm sorry the newbie question but should I create this new issue?
I'm just a novice tinkering around HBase looking for an opportunity to
contribute to such a great project (and, yes, I'm also looking for a project
for Google Summer of Code).
Thank you
On Sun, Mar 8, 2009 at 6:17 PM, Guilherme Mauro Germoglio Barbosa (JIRA) <
jira@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680016#action_12680016]
>
> Guilherme Mauro Germoglio Barbosa commented on HBASE-1094:
> ----------------------------------------------------------
>
> I think so.
>
> Also, I've edited HConnectionManager in order to use all HBaseConfiguration
> instead of only HBASE_DIR as key on the HBASE_INSTANCES map. To do so, I've
> implemented HBaseConfiguration.hashcode() -- just like a HashMap.hashCode()
> is implemented -- and all tests are green with these two modifications. So,
> I can upload the patch as soon as the issue is created.
>
> > Region Server throw NotServingRegionException when batchUpdate irredicly
> (After a large amount of file inserts)
> >
> ---------------------------------------------------------------------------------------------------------------
> >
> > Key: HBASE-1094
> > URL: https://issues.apache.org/jira/browse/HBASE-1094
> > Project: Hadoop HBase
> > Issue Type: Bug
> > Components: client
> > Affects Versions: 0.18.0, 0.18.1
> > Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2
> RegionServers
> > Reporter: Yossi Ittach
> > Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG,
> hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log,
> RSbugLog.log
> >
> >
> > After a large amount of file (~400,000) , once in a while during rapid
> inserts RegionServer returns NotServingRegionException when doing
> batchUpdate.
> > A restart of the client usually fixes the problem , but it happens again
> after a while.
> > LogFiles excerpts:
> > *RegionServer*
> > 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 1333
> > 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 1334
> > 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333
> size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> > 2008-12-16 09:17:14,439 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> > 2008-12-16 09:17:14,441 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> > 2008-12-16 09:17:14,461 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
> disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,461 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,461 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,461 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,462 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
> on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,462 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for
> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region
> memcache size 236.3k
> > 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 0
> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 1
> > 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.211:50010
> > 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0
> size:65557 offsetInBlock:0 lastPacketInBlock:false
> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 1
> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 2
> > 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1
> size:65557 offsetInBlock:65024 lastPacketInBlock:false
> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 2
> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 3
> > 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2
> size:65557 offsetInBlock:130048 lastPacketInBlock:false
> > 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3
> size:49273 offsetInBlock:195072 lastPacketInBlock:true
> > 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 1
> > 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 2
> > 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 3
> > 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_4611894842462358439_1544
> > 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.211:50010
> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0
> size:276 offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_-6152926948595840609_1544
> > 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/obde_content/info/5474929915463365960:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38
> offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_2767846628464404081_1545
> > 2008-12-16 09:17:14,644 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: Added
> /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56
> entries, sequence id 235152, data size 236.3k, file size 238.2k
> > 2008-12-16 09:17:14,644 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for
> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence
> id=235152, compaction requested=true
> > 2008-12-16 09:17:14,644 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> > 2008-12-16 09:17:14,644 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: closed
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 60020, call batchUpdate([B@552a2a7d, row =>
> f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0
> size:84 offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_-4579777298287321197_1547
> > 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 60020, call batchUpdate([B@18084038, row =>
> hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 60020, call batchUpdate([B@74da3b58, row =>
> rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 60020, call batchUpdate([B@51575d48, row =>
> aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 60020, call batchUpdate([B@63442ff5, row =>
> iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 60020, call batchUpdate([B@59eb5159, row =>
> t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 60020, call batchUpdate([B@132fa7c8, row =>
> Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 1333
> > 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 60020, call batchUpdate([B@4696de68, row =>
> Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 60020, call batchUpdate([B@16a3f072, row =>
> gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 1334
> > 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334
> size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> > 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 60020, call batchUpdate([B@171591e3, row =>
> ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 60020, call batchUpdate([B@452719a0, row =>
> bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 60020, call batchUpdate([B@6a76000a, row =>
> XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 60020, call batchUpdate([B@2b753bb9, row =>
> jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 60020, call batchUpdate([B@7b4286a2, row =>
> VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 1335
> > 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0
> size:84 offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_-1532247345865982631_1549
> > 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 60020, call batchUpdate([B@3f229bc1, row =>
> WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84
> offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_9044590686588152812_1551
> > 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 60020, call batchUpdate([B@74d3776e, row =>
> hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0
> size:84 offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_-3568743239673676944_1553
> > 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 60020, call batchUpdate([B@366480d7, row =>
> j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 60020, call batchUpdate([B@24af9995, row =>
> z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0
> size:84 offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_-1561317920938610537_1555
> > 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 60020, call batchUpdate([B@46823b18, row =>
> uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 60020, call batchUpdate([B@4718f92, row =>
> VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > org.apache.hadoop.hbase.NotServingRegionException: Region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient:
> /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010:
> masked=rwxr-xr-x
> > 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 0
> > 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
> new block
> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.102:50010
> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
> MY.IP.213:50010
> > 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting
> to MY.IP.102:50010
> > 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf
> size 131071
> > 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84
> offsetInBlock:0 lastPacketInBlock:true
> > 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> received ack for seqno 0
> > 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing
> old block blk_5499006216568952565_1557
> > 2008-12-16 09:17:16,081 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Opening region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> > 2008-12-16 09:17:16,117 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010,
> isReference=true, sequence id=207793
> > 2008-12-16 09:17:16,126 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010,
> isReference=true, sequence id=235152
> > 2008-12-16 09:17:16,134 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010,
> isReference=true, sequence id=235051
> > 2008-12-16 09:17:16,135 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore
> 1010339458/obde_content, max sequence id 235152
> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new decompressor
> > 2008-12-16 09:17:16,157 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> > 2008-12-16 09:17:16,159 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
> disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
> region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
> on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> > 2008-12-16 09:17:16,159 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: closed
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> > 2008-12-16 09:17:16,159 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Opening region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> > 2008-12-16 09:17:16,185 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010,
> isReference=true, sequence id=207793
> > 2008-12-16 09:17:16,193 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010,
> isReference=true, sequence id=235152
> > 2008-12-16 09:17:16,202 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: loaded
> /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010,
> isReference=true, sequence id=235051
> > 2008-12-16 09:17:16,203 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore
> 2015194550/obde_content, max sequence id 235152
> > 2008-12-16 09:17:16,229 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> > 2008-12-16 09:17:16,231 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes
> disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for
> region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding
> on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,231 DEBUG
> org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> > 2008-12-16 09:17:16,231 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: closed
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> > 2008-12-16 09:17:16,245 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up
> /hbase/my_table/1608250010/splits true
> > 2008-12-16 09:17:16,247 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in
> table locations for row <> and tableName .META.: location server
> MY.IP.102:60020, location region name .META.,,1
> > 2008-12-16 09:17:16,249 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
> updated, and report to master all successful. Old region=REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
> 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false',
> LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460,
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> > 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 7 on 60020, call batchUpdate([B@4d2637c2, row =>
> ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > org.apache.hadoop.hbase.NotServingRegionException:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 60020, call batchUpdate([B@816c920, row =>
> bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > org.apache.hadoop.hbase.NotServingRegionException:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> > at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> > 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk packet full seqno 1335
> > 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient:
> DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335
> size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> > 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
> writeChunk allocating new packet 1336
> > 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 8 on 60020, call batchUpdate([B@fee5806, row =>
> o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1)
> from MY.IP.212:46198: error:
> org.apache.hadoop.hbase.NotServingRegionException:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > *Master:*
> > 2008-12-16 09:15:31,712 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE =>
> 1229433746854
> > 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:15:46,728 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:16:01,743 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:16:16,760 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:16:31,775 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE =>
> 1229433746854
> > 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:16:46,791 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:17:01,807 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:17:16,823 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2,
> Avg Load: 2.0
> > 2008-12-16 09:17:16,902 INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from
> MY.IP.102:60020
> > 2008-12-16 09:17:19,826 INFO
> org.apache.hadoop.hbase.master.RegionManager: assigning region
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> > 2008-12-16 09:17:19,828 INFO
> org.apache.hadoop.hbase.master.RegionManager: assigning region
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> > 2008-12-16 09:17:22,833 INFO
> org.apache.hadoop.hbase.master.ServerManager: Received
> MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> from MY.IP.213:60020
> > 2008-12-16 09:17:22,833 INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> > 2008-12-16 09:17:22,833 INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> > 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Main processing loop: PendingOpenOperation from MY.IP.213:60020
> > 2008-12-16 09:17:22,833 INFO
> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> > 2008-12-16 09:17:22,833 DEBUG
> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
> 1, onlineMetaRegions.size(): 1
> > 2008-12-16 09:17:22,833 INFO
> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with
> startcode 1229433746880 and server MY.IP.213:60020
> > 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Main processing loop: PendingOpenOperation from MY.IP.213:60020
> > 2008-12-16 09:17:22,835 INFO
> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> > 2008-12-16 09:17:22,835 DEBUG
> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
> 1, onlineMetaRegions.size(): 1
> > 2008-12-16 09:17:22,835 INFO
> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with
> startcode 1229433746880 and server MY.IP.213:60020
> > 2008-12-16 09:17:31,845 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
> 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE =>
> 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.213:60020', STARTCODE => 1229433746880
> > 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
> 1229433746880
> > 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:17:46,864 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:18:01,879 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:18:16,895 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:18:31,912 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
> true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
> IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
> 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE =>
> 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.213:60020', STARTCODE => 1229433746880
> > 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
> 1229433746880
> > 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to
> my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> > 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
> Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because
> daughter splits no longer hold references
> > 2008-12-16 09:18:37,858 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://
> be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> > 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:18:46,927 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:19:01,943 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:19:16,960 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:19:31,976 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.213:60020', STARTCODE => 1229433746880
> > 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
> 1229433746880
> > 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:19:46,992 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:20:02,006 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:20:17,022 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:20:32,037 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020}
> > 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
> STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
> TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
> FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
> 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
> => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
> => 1229433746854
> > 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
> 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
> 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
> 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
> IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
> VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
> 'MY.IP.213:60020', STARTCODE => 1229433746880
> > 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner REGION => {NAME =>
> 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
> 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
> {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
> [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
> LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
> COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
> 1229433746880
> > 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
> startKey: <>, server: MY.IP.102:60020} complete
> > 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner:
> all meta regions scanned
> > 2008-12-16 09:20:47,052 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:21:02,069 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:21:17,084 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
> > 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020}
> > 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
> ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
> 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
> => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
> 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
> => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
> BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
> 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> > 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
> startKey: <>, server: MY.IP.213:60020} complete
> > 2008-12-16 09:21:32,100 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2,
> Avg Load: 3.0
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Guilherme
msn: guigermoglio@hotmail.com
homepage: http://germoglio.googlepages.com
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660150#action_12660150 ]
Jean-Daniel Cryans commented on HBASE-1094:
-------------------------------------------
bq. to the best of my understanding , the consecutive requests in the HRS are not the same request - notice it's a different row each time ,
Ah yes I overlooked that, with those crazy row names it's hard to follow ;) You should just keep the HBase DEBUG, not the Hadoop stuff so that the logs contain only what we need.
bq. What do you think happens here? is it because it's in the middle of a split (Aren't the request to a splitted region frozen until the split is over?) or is it the client using stale data (that somehow wasn't refreshed on time)? If so , wasn't it so suppose to update after the first NSRE ?
Yes the splits block the updates and after a NSRE the client cache refreshes. If the split is taking too long, META does not get updated and the client still gets the same information. Without your client log and your clean HRS log, it's really hard to get a hold of what's going on. Maybe it's because of an overwhelmed hardware (see my question in my last comment regards this subject).
bq. Will increasing the hbase.client.rpc.maxattempts do the trick?
You mean the hbase.client.retries.number? Yes, it should help along with hbase.client.pause.
Keep up the good work Yossi!
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661916#action_12661916 ]
yossale edited comment on HBASE-1094 at 1/8/09 2:13 AM:
-------------------------------------------------------------
Finally happened again
As you requested , I've got the stuck RS dump and the Master dump. (attached)
Also attached are the MasterUI and the RSUI. Notice both doesn't have the Region specified in the error , but have one that is very similar in name (I don't know if this is just a coincidence or not)
I've found these lines in an old master log file (from 1/109) - this is the lost region:
2009-01-01 16:08:04,169 DEBUG org.apache.hadoop.hbase.master.BaseScanner: obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230844034165 no longer has references to obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740
2009-01-01 16:08:04,171 DEBUG org.apache.hadoop.hbase.master.BaseScanner: obde_documentEntries,13k7Nop1AFBjYnwLf6p9KQ==,1230844034165 no longer has references to obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740
2009-01-01 16:08:04,172 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740 because daughter splits no longer hold references
2009-01-01 16:08:04,172 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/obde_documentEntries/318630300
* The console error: *
2009-01-08 03:31:57,419 ERROR [pool-3-thread-1] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@59653bcb
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740, row '1QGcnGSzaWZRtT5FcsTwwA==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
*The master log from that time:*
2009-01-08 03:30:44,890 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,w4eULkuoDVg9wtidBiSwHQ==,1230546207170', STARTKEY => 'w4eULkuoDVg9wtidBiSwHQ==', ENDKEY => 'xOZ+gfzITX5kg242uCQ==', ENCODED => 1259721264, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,891 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,xOZ+gfzITX5kg242uCQ==,1230546225176', STARTKEY => 'xOZ+gfzITX5kg242uCQ==', ENDKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENCODED => 1441754713, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.200:60020', STARTCODE => 1230815082100
2009-01-08 03:30:44,942 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,yhiJBIFf8OObcxlCY2qgsw==,1230546225176', STARTKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENDKEY => '', ENCODED => 552003722, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020} complete
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2009-01-08 03:30:47,486 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:30:58,225 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:02,502 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:17,522 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:29,074 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:32,539 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:41,728 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020}
2009-01-08 03:31:41,737 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:41,738 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020} complete
2009-01-08 03:31:44,841 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020}
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,,1230824142963', STARTKEY => '', ENDKEY => '+frhmXa3XDL4Hk4La1aUw==', ENCODED => 1237218248, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,+frhmXa3XDL4Hk4La1aUw==,1230824142963', STARTKEY => '+frhmXa3XDL4Hk4La1aUw==', ENDKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENCODED => 152138444, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230844034165', STARTKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENDKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENCODED => 311750939, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,13k7Nop1AFBjYnwLf6p9KQ==,1230844034165', STARTKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENDKEY => '1kqulDKTmT8Doi0Axzv2gA==', ENCODED => 1338919791, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
The RegionServer log from that time:
2009-01-08 03:31:57,188 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call get([B@293aa2ad, [B@488e2100, [B@6b18ac36, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,189 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call get([B@4c78aeed, [B@72470d4, [B@696b71f5, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,190 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call get([B@36a3a811, [B@326f101d, [B@198af9c9, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
was (Author: yossale):
Finally happened again
As you requested , I've got the stuck RS dump and the Master dump. (attached)
Also attached are the MasterUI and the RSUI. Notice both doesn't have the Region specified in the error , but have one that is very similar in name (I don't know if this is just a coincidence or not)
*The console error: *
2009-01-08 03:31:57,419 ERROR [pool-3-thread-1] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@59653bcb
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740, row '1QGcnGSzaWZRtT5FcsTwwA==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
*The master log from that time:*
2009-01-08 03:30:44,890 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,w4eULkuoDVg9wtidBiSwHQ==,1230546207170', STARTKEY => 'w4eULkuoDVg9wtidBiSwHQ==', ENDKEY => 'xOZ+gfzITX5kg242uCQ==', ENCODED => 1259721264, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,891 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,xOZ+gfzITX5kg242uCQ==,1230546225176', STARTKEY => 'xOZ+gfzITX5kg242uCQ==', ENDKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENCODED => 1441754713, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.200:60020', STARTCODE => 1230815082100
2009-01-08 03:30:44,942 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,yhiJBIFf8OObcxlCY2qgsw==,1230546225176', STARTKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENDKEY => '', ENCODED => 552003722, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020} complete
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2009-01-08 03:30:47,486 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:30:58,225 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:02,502 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:17,522 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:29,074 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:32,539 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:41,728 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020}
2009-01-08 03:31:41,737 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:41,738 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020} complete
2009-01-08 03:31:44,841 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020}
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,,1230824142963', STARTKEY => '', ENDKEY => '+frhmXa3XDL4Hk4La1aUw==', ENCODED => 1237218248, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,+frhmXa3XDL4Hk4La1aUw==,1230824142963', STARTKEY => '+frhmXa3XDL4Hk4La1aUw==', ENDKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENCODED => 152138444, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230844034165', STARTKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENDKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENCODED => 311750939, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,13k7Nop1AFBjYnwLf6p9KQ==,1230844034165', STARTKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENDKEY => '1kqulDKTmT8Doi0Axzv2gA==', ENCODED => 1338919791, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
The RegionServer log from that time:
2009-01-08 03:31:57,188 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call get([B@293aa2ad, [B@488e2100, [B@6b18ac36, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,189 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call get([B@4c78aeed, [B@72470d4, [B@696b71f5, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,190 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call get([B@36a3a811, [B@326f101d, [B@198af9c9, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660154#action_12660154 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
10x , I'm trying - This bug is driving me insane ;)
>> Maybe it's because of an overwhelmed hardware (see my question in my last comment regards this subject).
All the machines (3 RS+DN , 1 HBM&HNN) are Intel dual cores with 4G Ram , 80GB disk , and really weren't too busy (I remember checking that , but I can't give you actual figures)
>> You mean the hbase.client.retries.number? Yes, it should help along with hbase.client.pause
Actually I meant the maxRPCAttempts (in the HConnectionManager : "this.maxRPCAttempts = conf.getInt("hbase.client.rpc.maxattempts", 1);")
>>Without your client log and your clean HRS log, it's really hard to get a hold of what's going on
I'll try to keep everything next time it fails. However - What actually happens in my console is exactly as depicted here above: I get the NSRE excpetion on a specific file with this Exception:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916, row 'UQU17V6ogCy6Dl+0gQcUw==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
and then the system just skip (other RS still work). What bothers me the most is that from that moment on , all calls for that region fails (for hours!) and after I restart the client - everything works again. So this problem persist even after the region split is way over...
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665354#action_12665354 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
Jean-Daniel , is there something else that I can get to help fixing this issue?
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660134#action_12660134 ]
yossale edited comment on HBASE-1094 at 12/31/08 11:50 AM:
----------------------------------------------------------------
Nope , we haven't touched the pause time.
to the best of my understanding , the consecutive requests in the HRS are not the same request - notice it's a different row each time , but they all get the same NSRE reply. I'm trying to recreate the same error again (it happens sporadically , so I can't promise a date).
What do you think happens here? is it because it's in the middle of a split (Aren't the request to a splitted region frozen until the split is over?) or is it the client using stale data (that somehow wasn't refreshed on time)? If so , wasn't it so suppose to update after the first NSRE ?
Will increasing the hbase.client.rpc.maxattempts do the trick?
I'll try to find the exact console log again and post it
was (Author: yossale):
Nope , we haven't touched the pause time.
to the best of my understanding , the consecutive requests in the HRS are not the same request - notice it's a different row each time , but they all get the same NSRE reply. I'm trying to recreate the same error again (it happens sporadically , so I can't promise a date).
What do you think happens here? is it because it's in the middle of a split (Aren't the request to a splitted region frozen until the split is over?) or is it the client using stale data (that somehow wasn't refreshed on time)? If so , wasn't it so suppose to update after the first NSRE ?
I'll try to find the exact console log again and post it
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660156#action_12660156 ]
Jean-Daniel Cryans commented on HBASE-1094:
-------------------------------------------
Okay the hardware is alright, cluster is small but you only have 1 client inserting rows so it shouldn't be that bad.
Regards hbase.client.rpc.maxattempts, it's weird to have a conf missing from hbase-default.xml, care to open a JIRA for this? Thx. But upping it would not affect your situation.
That thing with the client holding of the HRS, it's very weird. Next time it happens, try to get a Thread Dump from the stuck region server. You can get it from the HRS web UI or by using jstack on the machine.
Thanks.
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yossi Ittach updated HBASE-1094:
--------------------------------
Comment: was deleted
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Guilherme Mauro Germoglio Barbosa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680016#action_12680016 ]
Guilherme Mauro Germoglio Barbosa commented on HBASE-1094:
----------------------------------------------------------
I think so.
Also, I've edited HConnectionManager in order to use all HBaseConfiguration instead of only HBASE_DIR as key on the HBASE_INSTANCES map. To do so, I've implemented HBaseConfiguration.hashcode() -- just like a HashMap.hashCode() is implemented -- and all tests are green with these two modifications. So, I can upload the patch as soon as the issue is created.
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660085#action_12660085 ]
yossale edited comment on HBASE-1094 at 12/31/08 7:41 AM:
---------------------------------------------------------------
Hey Jean-Daniel,
>>Did you open this JIRA when seeing the NotServingRegionException in the HRS log
Nope , I started it because I'm recieving the NSRE in my console , under a RetriesExhaustedException
>>When a region gets closed, the clients with a stale cache will still try to use it unless
>>they get a NotServingRegionException and they will show up in the HRS log
I figured that this is how it should happen , but the problem is that It doesn't work:
In my console I get an RetriesExhaustedException (Originiated in (I think) the HConnectionManager@getRegionServerWithRetries), with a list of NSRE's - and since then , any attempt to write to the same region fails.
Also - this happens after a substantial amount of files - we've already had many region splits before it happened - so it doesn't seem to be so straight forward
>> So, how are you inserting the files in HBase?
HBase 0.18.1 on Hadoop 0.18.1 , 3 DN&RS + 1HBM & 1HNN
Using hTable.commit(batchUpdate); (Same as the API example...)
>> Many java clients?
Just one currently
>> Using Thrift? Nope
>> many clients? How do they react when getting this exception?
Currently 1 client - catches the exception and skips this file (Also tried re-entering the same file - didn't work)
>>Client log:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> Looked for the region in the shell , and got this:
hbase(main):002:0> get '.META.', 'obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802'
08/12/30 07:48:12 DEBUG client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.216:60020, location region name .META.,,1
COLUMN CELL
historian:assignment timestamp=1230640777222, value=Region assigned to server 192.168.252.200:60020
historian:compaction timestamp=1230640790641, value=Region compaction completed in 5sec
historian:open timestamp=1230640780231, value=Region opened on server : My_Server_Name
historian:split timestamp=1230551637944, value=Region split from : obde_documentEntries,RRs+UqZu
I30ot7HG+X57FQ==,1230477626688
info:regioninfo timestamp=1230551637946, value=REGION => {NAME => 'obde_documentEntries,RRs+UqZuI
30ot7HG+X57FQ==,1230551636802', STARTKEY => 'RRs+UqZuI30ot7HG+X57FQ==', ENDKEY =>
'SSv6VcCov7wlee67ZjNlXQ==', ENCODED => 1710096259, TABLE => {{NAME => 'obde_docu
mentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde
_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCA
CHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}
info:server timestamp=1230640780230, value=192.168.252.200:60020
info:serverstartcode timestamp=1230640780230, value=1230640763821
7 row(s) in 0.0270 seconds
was (Author: yossale):
Hey Jean-Daniel,
>>Did you open this JIRA when seeing the NotServingRegionException in the HRS log
Nope , I started it because I'm recieving the NSRE in my console , under a RetriesExhaustedException
>>When a region gets closed, the clients with a stale cache will still try to use it unless >>they get a NotServingRegionException and they will show up in the HRS log
I figured that this is how it should happen , but the problem is that It doesn't work:
In my console I get an RetriesExhaustedException (Originiated in (I think) the HConnectionManager@getRegionServerWithRetries), with a list of NSRE's - and since then , any attempt to write to the same region fails.
>> So, how are you inserting the files in HBase?
HBase 0.18.1 on Hadoop 0.18.1 , 3 DN&RS + 1HBM & 1HNN
>> Many java clients?
Just one currently
>> Using Thrift? Nope
>> many clients? How do they react when getting this exception?
Currently 1 client - catches the exception and skips this file (Also tried re-entering the same file - didn't work)
>>Client log:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> Looked for the region in the shell , and got this:
hbase(main):002:0> get '.META.', 'obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802'
08/12/30 07:48:12 DEBUG client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.216:60020, location region name .META.,,1
COLUMN CELL
historian:assignment timestamp=1230640777222, value=Region assigned to server 192.168.252.200:60020
historian:compaction timestamp=1230640790641, value=Region compaction completed in 5sec
historian:open timestamp=1230640780231, value=Region opened on server : My_Server_Name
historian:split timestamp=1230551637944, value=Region split from : obde_documentEntries,RRs+UqZu
I30ot7HG+X57FQ==,1230477626688
info:regioninfo timestamp=1230551637946, value=REGION => {NAME => 'obde_documentEntries,RRs+UqZuI
30ot7HG+X57FQ==,1230551636802', STARTKEY => 'RRs+UqZuI30ot7HG+X57FQ==', ENDKEY =>
'SSv6VcCov7wlee67ZjNlXQ==', ENCODED => 1710096259, TABLE => {{NAME => 'obde_docu
mentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde
_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCA
CHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}
info:server timestamp=1230640780230, value=192.168.252.200:60020
info:serverstartcode timestamp=1230640780230, value=1230640763821
7 row(s) in 0.0270 seconds
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yossi Ittach updated HBASE-1094:
--------------------------------
Attachment: 1-Bug-egionServer2.JPG
1-Bug-Master.JPG
This is the MasterUI and the RSUI. Notice both doesn't have the Region specified in the error , but have one that is very similar in name (I don't know if this is just a coincidence or not)
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yossi Ittach updated HBASE-1094:
--------------------------------
Attachment: RSbugLog.log
HMBugLog.log
part of the log files
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660134#action_12660134 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
Nope , we haven't touched the pause time.
to the best of my understanding , the consecutive requests in the HRS are not the same request - notice it's a different row each time , but they all get the same NSRE reply. I'm trying to recreate the same error again (it happens sporadically , so I can't promise a date).
What do you think happens here? is it because it's in the middle of a split (Aren't the request to a splitted region frozen until the split is over?) or is it the client using stale data (that somehow wasn't refreshed on time)? If so , wasn't it so suppose to update after the first NSRE ?
I'll try to find the exact console log again and post it
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660154#action_12660154 ]
yossale edited comment on HBASE-1094 at 12/31/08 2:09 PM:
---------------------------------------------------------------
10x , I'm trying - This bug is driving me insane ;)
>> Maybe it's because of an overwhelmed hardware (see my question in my last comment regards this subject).
All the machines (3 RS+DN , 1 HBM&HNN) are Intel dual cores with 4G Ram , 80GB disk , and really weren't too busy (I remember checking that , but I can't give you actual figures)
>> You mean the hbase.client.retries.number? Yes, it should help along with hbase.client.pause
Actually I meant the maxRPCAttempts (in the HConnectionManager : "this.maxRPCAttempts = conf.getInt("hbase.client.rpc.maxattempts", 1);")
>>Without your client log and your clean HRS log, it's really hard to get a hold of what's going on
I'll try to keep everything next time it fails. - I've attached some console log I found . It's pretty repetitive , but notice that same problem occurs on 2 different region servers (213 & 216) .
After a NSRE the system just skip What bothers me the most is that from that moment on , all calls for that region fails (for hours!) - So this problem persist even long after the region split is way over... and after I restart the client - everything works again.
Console logs:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,s2WGdnjkfzOQ36wsxZFsGw==,1230546252886
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:55,056 ERROR [pool-3-thread-6] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.216:60020 for region obde_documentEntries,AxIfrQtyK6ln3gAx7Z6lg==,1230557004812, row 'BXabo2C5jyl9a0mC1XtlOg==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,AxIfrQtyK6ln3gAx7Z6lg==,1230557004812
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916, row 'UQU17V6ogCy6Dl+0gQcUw==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:222)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:55,262 ERROR [pool-3-thread-4] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,ukzPch1iK2Z2ALoHEiVA==,1230546207170, row 'vE2+lFgz1znEI7sSpTr4Xw==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,ukzPch1iK2Z2ALoHEiVA==,1230546207170
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:55,611 ERROR [pool-3-thread-6] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.216:60020 for region obde_documentEntries,KazxMIqswPDW3U4XJZNa3w==,1230560572107, row 'L78TGyh0zoNNw7JRg1XnJA==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,KazxMIqswPDW3U4XJZNa3w==,1230560572107
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:222)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:55,734 ERROR [pool-3-thread-4] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916, row 'TUtfp4qZwWVZK3NEmYpAA==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:56,635 ERROR [pool-3-thread-4] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,irBHbhckwm0G71ycf3K1IQ==,1230546338725, row 'jFyN0kMQHJgf5UIyiEYk5w==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,irBHbhckwm0G71ycf3K1IQ==,1230546338725
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2008-12-30 07:48:57,350 ERROR [pool-3-thread-6] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@5b9bb7ff
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,ukzPch1iK2Z2ALoHEiVA==,1230546207170, row 'vsGhQwtevCAOmYrevAPYQ==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,ukzPch1iK2Z2ALoHEiVA==,1230546207170
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:222)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
was (Author: yossale):
10x , I'm trying - This bug is driving me insane ;)
>> Maybe it's because of an overwhelmed hardware (see my question in my last comment regards this subject).
All the machines (3 RS+DN , 1 HBM&HNN) are Intel dual cores with 4G Ram , 80GB disk , and really weren't too busy (I remember checking that , but I can't give you actual figures)
>> You mean the hbase.client.retries.number? Yes, it should help along with hbase.client.pause
Actually I meant the maxRPCAttempts (in the HConnectionManager : "this.maxRPCAttempts = conf.getInt("hbase.client.rpc.maxattempts", 1);")
>>Without your client log and your clean HRS log, it's really hard to get a hold of what's going on
I'll try to keep everything next time it fails. However - What actually happens in my console is exactly as depicted here above: I get the NSRE excpetion on a specific file with this Exception:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916, row 'UQU17V6ogCy6Dl+0gQcUw==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,TUpPkdJDxYdsWCxlMhxJLQ==,1230551614916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
and then the system just skip (other RS still work). What bothers me the most is that from that moment on , all calls for that region fails (for hours!) and after I restart the client - everything works again. So this problem persist even after the region split is way over...
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679919#action_12679919 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
A very interesting observation - I'll try to check if it works for me too !
Vale et me ama
Yossi
On Sat, Mar 7, 2009 at 10:26 PM, Guilherme Mauro Germoglio Barbosa (JIRA) <
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660083#action_12660083 ]
Jean-Daniel Cryans commented on HBASE-1094:
-------------------------------------------
Yossi,
What I'm seeing is is a completely normal system. When a region gets closed, the clients with a stale cache will still try to use it unless they get a NotServingRegionException and they will show up in the HRS log. Also you say that a client restart solves the problem, which leads me to believe that the problem is not there.
So, how are you inserting the files in HBase? Many java clients? Using Thrift? How many clients? How do they react when getting this exception? You should also be getting some logs in your client, can we see it (make sure they also are in DEBUG just like your cluster)? Did you open this JIRA when seeing the NotServingRegionException in the HRS log?
Thanks
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679926#action_12679926 ]
stack commented on HBASE-1094:
------------------------------
Guilherme: Not expected behavior. Should we make a new issue to fix what you've observed? Yossi, doing as Guilherme suggests, do things work better for you?
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660087#action_12660087 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
Hey Jean-Daniel,
HRS log
Nope , I started it because I'm recieving the NSRE in my console , under a
RetriesExhaustedException
to use it unless >>they get a NotServingRegionException and they will show
up in the HRS log
I figured that this is how it should happen , but the problem is that It
doesn't work:
In my console I get an RetriesExhaustedException (Originiated in originated
in (I think) the HConnectionManager@getRegionServerWithRetries), with a list
of NSRE's - and since then , any attempt to write to the same region fails.
HBase 0.18.1 on Hadoop 0.18.1 , 3 DN&RS + 1HBM & 1HNN
Just one currently
Currently 1 client - catches the exception and skips this file (Also tried
re-entering the same file - didn't work)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
hbase(main):002:0> get '.META.',
'obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802'
08/12/30 07:48:12 DEBUG client.HConnectionManager$TableServers: Cache hit in
table locations for row <> and tableName .META.: location server
192.168.252.216:60020, location region name .META.,,1
COLUMN CELL
historian:assignment timestamp=1230640777222, value=Region assigned
to server 192.168.252.200:60020
historian:compaction timestamp=1230640790641, value=Region
compaction completed in 5sec
historian:open timestamp=1230640780231, value=Region opened on
server : ofl2.nydc1.outbrain.com
historian:split timestamp=1230551637944, value=Region split
from : obde_documentEntries,RRs+UqZu
I30ot7HG+X57FQ==,1230477626688
info:regioninfo timestamp=1230551637946, value=REGION => {NAME
=> 'obde_documentEntries,RRs+UqZuI
30ot7HG+X57FQ==,1230551636802', STARTKEY =>
'RRs+UqZuI30ot7HG+X57FQ==', ENDKEY =>
'SSv6VcCov7wlee67ZjNlXQ==', ENCODED =>
1710096259, TABLE => {{NAME => 'obde_docu
mentEntries', IS_ROOT => 'false', IS_META =>
'false', FAMILIES => [{NAME => 'obde
_content', BLOOMFILTER => 'false', IN_MEMORY =>
'false', VERSIONS => '3', BLOCKCA
CHE => 'false', LENGTH => '2147483647', TTL =>
'-1', COMPRESSION => 'NONE'}]}}
info:server timestamp=1230640780230, value=
192.168.252.200:60020
info:serverstartcode timestamp=1230640780230, value=1230640763821
7 row(s) in 0.0270 seconds
Vale et me ama
Yossi
On Wed, Dec 31, 2008 at 5:16 PM, Jean-Daniel Cryans (JIRA)
<ji...@apache.org>wrote:
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660085#action_12660085 ]
Yossi Ittach commented on HBASE-1094:
-------------------------------------
Hey Jean-Daniel,
>>Did you open this JIRA when seeing the NotServingRegionException in the HRS log
Nope , I started it because I'm recieving the NSRE in my console , under a RetriesExhaustedException
>>When a region gets closed, the clients with a stale cache will still try to use it unless >>they get a NotServingRegionException and they will show up in the HRS log
I figured that this is how it should happen , but the problem is that It doesn't work:
In my console I get an RetriesExhaustedException (Originiated in (I think) the HConnectionManager@getRegionServerWithRetries), with a list of NSRE's - and since then , any attempt to write to the same region fails.
>> So, how are you inserting the files in HBase?
HBase 0.18.1 on Hadoop 0.18.1 , 3 DN&RS + 1HBM & 1HNN
>> Many java clients?
Just one currently
>> Using Thrift? Nope
>> many clients? How do they react when getting this exception?
Currently 1 client - catches the exception and skips this file (Also tried re-entering the same file - didn't work)
>>Client log:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>> Looked for the region in the shell , and got this:
hbase(main):002:0> get '.META.', 'obde_documentEntries,RRs+UqZuI30ot7HG+X57FQ==,1230551636802'
08/12/30 07:48:12 DEBUG client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.216:60020, location region name .META.,,1
COLUMN CELL
historian:assignment timestamp=1230640777222, value=Region assigned to server 192.168.252.200:60020
historian:compaction timestamp=1230640790641, value=Region compaction completed in 5sec
historian:open timestamp=1230640780231, value=Region opened on server : My_Server_Name
historian:split timestamp=1230551637944, value=Region split from : obde_documentEntries,RRs+UqZu
I30ot7HG+X57FQ==,1230477626688
info:regioninfo timestamp=1230551637946, value=REGION => {NAME => 'obde_documentEntries,RRs+UqZuI
30ot7HG+X57FQ==,1230551636802', STARTKEY => 'RRs+UqZuI30ot7HG+X57FQ==', ENDKEY =>
'SSv6VcCov7wlee67ZjNlXQ==', ENCODED => 1710096259, TABLE => {{NAME => 'obde_docu
mentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde
_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCA
CHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}
info:server timestamp=1230640780230, value=192.168.252.200:60020
info:serverstartcode timestamp=1230640780230, value=1230640763821
7 row(s) in 0.0270 seconds
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661965#action_12661965 ]
Jean-Daniel Cryans commented on HBASE-1094:
-------------------------------------------
The region with a similiar name is a child of the region your client is searching. After a split, one of the 2 regions will retain the same start key but the ID (the numbers after that) will be different since it's kind of a timestamp.
You seem to have a case of leftovers in META. To make sure, do a scan of .META. in the shell on column family 'historian:' and look for a row key named "obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740". If so, delete that.
Regards your region server log, I don't think it's the good one since it says org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661916#action_12661916 ]
yossale edited comment on HBASE-1094 at 1/8/09 2:07 AM:
-------------------------------------------------------------
Finally happened again
As you requested , I've got the stuck RS dump and the Master dump. (attached)
Also attached are the MasterUI and the RSUI. Notice both doesn't have the Region specified in the error , but have one that is very similar in name (I don't know if this is just a coincidence or not)
*The console error: *
2009-01-08 03:31:57,419 ERROR [pool-3-thread-1] [com.outbrain.globals.io.filesystem.HBaseFeedEntries] (HBaseFeedEntries.java:130) - Could not commit to table: [B@59653bcb
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.252.213:60020 for region obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740, row '1QGcnGSzaWZRtT5FcsTwwA==', but failed after 1 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230485251740
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveToMechanisem(HBaseFeedEntries.java:125)
at com.outbrain.globals.io.filesystem.HBaseFeedEntries.saveTo(HBaseFeedEntries.java:102)
at com.outbrain.globals.io.filesystem.FeedEntriesFS.saveTo(FeedEntriesFS.java:157)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.updateDocuments(SourceCrawler.java:214)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.handleSource(SourceCrawler.java:89)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:52)
at com.outbrain.source_detector.sourceCrawler.SourceCrawler.call(SourceCrawler.java:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
*The master log from that time:*
2009-01-08 03:30:44,890 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,w4eULkuoDVg9wtidBiSwHQ==,1230546207170', STARTKEY => 'w4eULkuoDVg9wtidBiSwHQ==', ENDKEY => 'xOZ+gfzITX5kg242uCQ==', ENCODED => 1259721264, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,891 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,xOZ+gfzITX5kg242uCQ==,1230546225176', STARTKEY => 'xOZ+gfzITX5kg242uCQ==', ENDKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENCODED => 1441754713, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.200:60020', STARTCODE => 1230815082100
2009-01-08 03:30:44,942 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,yhiJBIFf8OObcxlCY2qgsw==,1230546225176', STARTKEY => 'yhiJBIFf8OObcxlCY2qgsw==', ENDKEY => '', ENCODED => 552003722, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020} complete
2009-01-08 03:30:44,943 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
2009-01-08 03:30:47,486 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:30:58,225 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:02,502 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:17,522 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:29,074 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server 192.168.252.213:60020, location region name .META.,,1
2009-01-08 03:31:32,539 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 63, Num Servers: 3, Avg Load: 21.0
2009-01-08 03:31:41,728 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020}
2009-01-08 03:31:41,737 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:41,738 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 192.168.252.216:60020} complete
2009-01-08 03:31:44,841 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: 192.168.252.213:60020}
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,,1230824142963', STARTKEY => '', ENDKEY => '+frhmXa3XDL4Hk4La1aUw==', ENCODED => 1237218248, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,865 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,+frhmXa3XDL4Hk4La1aUw==,1230824142963', STARTKEY => '+frhmXa3XDL4Hk4La1aUw==', ENDKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENCODED => 152138444, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.216:60020', STARTCODE => 1230815082235
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,0NCwYeyHUHMQUlwlhQhiWQ==,1230844034165', STARTKEY => '0NCwYeyHUHMQUlwlhQhiWQ==', ENDKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENCODED => 311750939, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
2009-01-08 03:31:44,866 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'obde_documentEntries,13k7Nop1AFBjYnwLf6p9KQ==,1230844034165', STARTKEY => '13k7Nop1AFBjYnwLf6p9KQ==', ENDKEY => '1kqulDKTmT8Doi0Axzv2gA==', ENCODED => 1338919791, TABLE => {{NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '3', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => '192.168.252.213:60020', STARTCODE => 1230815085374
The RegionServer log from that time:
2009-01-08 03:31:57,188 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call get([B@293aa2ad, [B@488e2100, [B@6b18ac36, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,Xll7oHIEuAXIGxMQn68onw==,1230549771985 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,189 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call get([B@4c78aeed, [B@72470d4, [B@696b71f5, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,7L08uqnO1h1SjsFbQHvYw==,1230815130454 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2009-01-08 03:31:57,190 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call get([B@36a3a811, [B@326f101d, [B@198af9c9, -1, -1) from 192.168.252.216:43548: error: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family on content: does not exist in region obde_documentEntries,UW4y+oARe5pOGYqAmDwDg==,1230551614916 in table {NAME => 'obde_documentEntries', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '3', TTL => '-1', COMPRESSION => 'NONE'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkColumn(HRegion.java:1776)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:1151)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1020)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
was (Author: yossale):
This is the MasterUI and the RSUI. Notice both doesn't have the Region specified in the error , but have one that is very similar in name (I don't know if this is just a coincidence or not)
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Yossi Ittach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yossi Ittach updated HBASE-1094:
--------------------------------
Attachment: hbase-regionserver-dump.out
hbase-master-dump.out
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Guilherme Mauro Germoglio Barbosa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679908#action_12679908 ]
Guilherme Mauro Germoglio Barbosa commented on HBASE-1094:
----------------------------------------------------------
Hi Yossi and Jean-Daniel,
I've just had a similar problem when evaluating HBase.
When I was running a simple stress test (many threads putting data into a single table), my client started to throw several org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server for region t,w45x D????|?v?-?,1236452295439, row 'x4q?1??t(??j?(G?.??', but failed after *1* attempts. The strange thing was that I was quite sure that I have set both hbase.client.retries.number and hbase.client.rpc.maxattempts to 10. So I thought there was no way to my client fail after only one attempt.
After further investigation, I think I've found where was the problem.
My client, just before starting the threads, invokes HBaseAdmin.checkHBaseAvailable(config). Although this method copies the HBaseConfiguration object and sets hbase.client.retries.number to 1 (see HBaseAdmin, line 751), it creates an HBaseAdmin object, which invokes HConnectionManager.getConnection(conf). Please notice that this conf is that with hbase.client.retries.number equals to 1. HConnectionManager.getConnection then creates a HConnection using this conf and puts it into a somekind of cache (see HConnectionManager, line 93). Then, the problem with my client is when my it creates a HTable object to be used by its threads it will invoke HConnectionManager.getConnection(conf) again (see HTable, line 109), this time with the correct Configuration object -- the one with hbase.client.retries.number equals to 10. However, when it checks the cache for a HConnection it finds one -- the one previously created by the HBaseAdmin object and using another Configuration object -- and returns it without creating a new one with the correct Configuration.
Well, I've fixed the client for my tests removing the line HBaseAdmin.checkHBaseAvailable(), however I don't think the behavior I've described is expected. What do you think, Jean-Daniel?
Thank you very much
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: 1-Bug-egionServer2.JPG, 1-Bug-Master.JPG, hbase-master-dump.out, hbase-regionserver-dump.out, HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1094) Region Server throw
NotServingRegionException when batchUpdate irredicly (After a large amount
of file inserts)
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660111#action_12660111 ]
Jean-Daniel Cryans commented on HBASE-1094:
-------------------------------------------
Oh so you did get the RetriesExhaustedException, that's better (would be nice to see the whole exception with the 10 NSRE and timestamps ). Relooking at the logs, I see that your client got the NSRE 10 times under 3 seconds which is not enough for the master to finish the splitting procedure it seems. First one was at 09:17:14,649 and last one at 09:17:16,416. This is weird, because we have a minimum of a 1 or 2 seconds retry then it's incremental. Did you change the hbase.client.pause value? If not, you could try putting it at 4000 instead of 2000.
Could you also describe the hardware you are using? I'd also like to know, when the inserts fail, if the machines are swapping/using 100% CPU.
Thx!
> Region Server throw NotServingRegionException when batchUpdate irredicly (After a large amount of file inserts)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1094
> URL: https://issues.apache.org/jira/browse/HBASE-1094
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.18.0, 0.18.1
> Environment: CentOS 64bit , Hadoop 0.18.1 , 5 DataNode + 2 RegionServers
> Reporter: Yossi Ittach
> Attachments: HMBugLog.log, RSbugLog.log
>
>
> After a large amount of file (~400,000) , once in a while during rapid inserts RegionServer returns NotServingRegionException when doing batchUpdate.
> A restart of the client usually fixes the problem , but it happens again after a while.
> LogFiles excerpts:
> *RegionServer*
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1333
> 2008-12-16 09:17:14,413 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1334
> 2008-12-16 09:17:14,414 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1333 size:65557 offsetInBlock:19507200 lastPacketInBlock:false
> 2008-12-16 09:17:14,439 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 2sec
> 2008-12-16 09:17:14,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,442 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits: masked=rwxr-xr-x
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,461 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,462 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395. Current region memcache size 236.3k
> 2008-12-16 09:17:14,463 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,484 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/data: masked=rwxr-xr-x
> 2008-12-16 09:17:14,505 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960/index: masked=rwxr-xr-x
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 0
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1
> 2008-12-16 09:17:14,525 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,526 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,527 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,530 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:0 size:65557 offsetInBlock:0 lastPacketInBlock:false
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 2
> 2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557 offsetInBlock:65024 lastPacketInBlock:false
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 2
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 3
> 2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557 offsetInBlock:130048 lastPacketInBlock:false
> 2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273 offsetInBlock:195072 lastPacketInBlock:true
> 2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1
> 2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 2
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 3
> 2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_4611894842462358439_1544
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.211:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-6152926948595840609_1544
> 2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/obde_content/info/5474929915463365960: masked=rwxr-xr-x
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_2767846628464404081_1545 wrote packet seqno:0 size:38 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_2767846628464404081_1545
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960 with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence id=235152, compaction requested=true
> 2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1608250010/obde_content
> 2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020, call batchUpdate([B@552a2a7d, row => f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-4579777298287321197_1547
> 2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@18084038, row => hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74da3b58, row => rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@51575d48, row => aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@63442ff5, row => iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:14,998 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@59eb5159, row => t5gRF1zQOrx27LTUS84ADA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,023 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@132fa7c8, row => Tu0dl1jFT2spZmEZundPoA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,045 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 1333
> 2008-12-16 09:17:15,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@4696de68, row => Skj4FHgWdrR+DSOppIaG6Q==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,237 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@16a3f072, row => gWd3yPCu6A9CxjBAhO3UPg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1334
> 2008-12-16 09:17:15,453 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1334 size:65557 offsetInBlock:19572224 lastPacketInBlock:false
> 2008-12-16 09:17:15,486 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020, call batchUpdate([B@171591e3, row => ddd8ecfGPwfgKzuLNMvOAw==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,498 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@452719a0, row => bJOTHv64AsNoFYsh7D0UyA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,508 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020, call batchUpdate([B@6a76000a, row => XVNFoDhT1NeKnT0YwhopHg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,548 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate([B@2b753bb9, row => jHxdShcit3A24oL3HHHgdg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,557 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@7b4286a2, row => VYUtICqregww41FxZUJkig==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,558 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1335
> 2008-12-16 09:17:15,659 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1532247345865982631_1549 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,661 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1532247345865982631_1549
> 2008-12-16 09:17:15,679 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,721 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/4722584128214377088.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,744 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,745 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@3f229bc1, row => WQXUl6gepHza8rVTmr8BSA==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,745 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,746 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,747 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,750 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_9044590686588152812_1551 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,752 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_9044590686588152812_1551
> 2008-12-16 09:17:15,774 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@74d3776e, row => hfCFR4B8pMF3mnY9mMnkDQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,817 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/6108448749728539444.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,841 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,842 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,848 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-3568743239673676944_1553 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,850 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-3568743239673676944_1553
> 2008-12-16 09:17:15,876 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,901 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate([B@366480d7, row => j4MGoh0tZvLe0bwYS5lqFQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,925 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@24af9995, row => z1Uyyyuot9H+G5kHIOfveg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,926 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/1010339458/obde_content/info/3431594424093212534.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:15,949 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:15,950 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:15,951 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:15,954 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_-1561317920938610537_1555 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:15,956 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_-1561317920938610537_1555
> 2008-12-16 09:17:15,981 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020, call batchUpdate([B@46823b18, row => uwp0i8x0qk8RVIjimK1Nfg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:15,983 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,005 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020, call batchUpdate([B@4718f92, row => VS0OVRXK4BnH9VUHB3rrSQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
> at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
> at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,030 DEBUG org.apache.hadoop.dfs.DFSClient: /hbase/my_table/1608250010/splits/2015194550/obde_content/info/4022324240389174248.1608250010: masked=rwxr-xr-x
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 0
> 2008-12-16 09:17:16,053 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating new block
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.102:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline = MY.IP.213:50010
> 2008-12-16 09:17:16,054 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to MY.IP.102:50010
> 2008-12-16 09:17:16,055 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size 131071
> 2008-12-16 09:17:16,058 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_5499006216568952565_1557 wrote packet seqno:0 size:84 offsetInBlock:0 lastPacketInBlock:true
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient received ack for seqno 0
> 2008-12-16 09:17:16,060 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old block blk_5499006216568952565_1557
> 2008-12-16 09:17:16,081 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458
> 2008-12-16 09:17:16,117 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/1405033884904780036.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,126 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/3431594424093212534.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,134 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/1010339458/obde_content/info/4722584128214377088.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,135 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 1010339458/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,149 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,156 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2008-12-16 09:17:16,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 is 235153
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460/1010339458 available
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1010339458/obde_content
> 2008-12-16 09:17:16,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
> 2008-12-16 09:17:16,159 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550
> 2008-12-16 09:17:16,185 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/215038473253378290.1608250010, isReference=true, sequence id=207793
> 2008-12-16 09:17:16,193 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/4022324240389174248.1608250010, isReference=true, sequence id=235152
> 2008-12-16 09:17:16,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore: loaded /hbase/my_table/2015194550/obde_content/info/6108448749728539444.1608250010, isReference=true, sequence id=235051
> 2008-12-16 09:17:16,203 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Loaded 3 file(s) in hstore 2015194550/obde_content, max sequence id 235152
> 2008-12-16 09:17:16,229 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 is 235153
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460/2015194550 available
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and cache flushes disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active scanners for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,231 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2015194550/obde_content
> 2008-12-16 09:17:16,231 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
> 2008-12-16 09:17:16,245 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Cleaned up /hbase/my_table/1608250010/splits true
> 2008-12-16 09:17:16,247 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit in table locations for row <> and tableName .META.: location server MY.IP.102:60020, location region name .META.,,1
> 2008-12-16 09:17:16,249 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}, new regions: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460, my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460. Split took 1sec
> 2008-12-16 09:17:16,291 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020, call batchUpdate([B@4d2637c2, row => ZEhP2cZMl+ovnXmpbN5UOQ==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,301 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate([B@816c920, row => bDPORkmlvmRIZH0yOux0Sg==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk packet full seqno 1335
> 2008-12-16 09:17:16,352 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer block blk_7750693789795884606_1536 wrote packet seqno:1335 size:65557 offsetInBlock:19637248 lastPacketInBlock:false
> 2008-12-16 09:17:16,356 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient writeChunk allocating new packet 1336
> 2008-12-16 09:17:16,416 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020, call batchUpdate([B@fee5806, row => o8MMWrSbpBj8hFPXSyhZ+g==, {column => obde_content:, value => '...'}, -1) from MY.IP.212:46198: error: org.apache.hadoop.hbase.NotServingRegionException: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> *Master:*
> 2008-12-16 09:15:31,712 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:15:31,878 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:15:31,908 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:31,909 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:15:37,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:15:37,817 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,818 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:15:37,819 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:15:46,728 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:01,743 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:16,760 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,775 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:16:31,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:16:31,920 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:31,922 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:16:37,804 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:16:37,826 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 4, Num Servers: 2, Avg Load: 2.0
> 2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395: [B@674c5b37 from MY.IP.102:60020
> 2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop: PendingOpenOperation from MY.IP.213:60020
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
> 2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2008-12-16 09:17:22,835 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with startcode 1229433746880 and server MY.IP.213:60020
> 2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:17:46,864 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:01,879 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:16,895 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,910 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:18:31,912 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:18:31,935 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:31,936 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:18:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:18:37,851 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,852 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,853 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:18:37,857 DEBUG org.apache.hadoop.hbase.master.BaseScanner: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 no longer has references to my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
> 2008-12-16 09:18:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 because daughter splits no longer hold references
> 2008-12-16 09:18:37,858 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://be4.nydc1.outbrain.com:54310/hbase/my_table/1608250010
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:18:37,891 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:18:46,927 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:01,943 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:16,960 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:31,921 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:19:31,949 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:31,950 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:19:31,976 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:19:37,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:19:37,861 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:19:37,862 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:19:37,863 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:19:46,992 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:02,006 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:17,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:31,931 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:20:31,955 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:31,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:20:32,037 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:20:37,848 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020}
> 2008-12-16 09:20:37,870 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395', STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:20:37,871 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED => 1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner REGION => {NAME => 'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE => 1229433746880
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of meta region {regionname: .META.,,1, startKey: <>, server: MY.IP.102:60020} complete
> 2008-12-16 09:20:37,872 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
> 2008-12-16 09:20:47,052 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:02,069 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:17,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
> 2008-12-16 09:21:31,942 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020}
> 2008-12-16 09:21:31,964 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME => 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1', BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
> 2008-12-16 09:21:31,965 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: MY.IP.213:60020} complete
> 2008-12-16 09:21:32,100 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 5, Num Servers: 2, Avg Load: 3.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.