You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (JIRA)" <ji...@apache.org> on 2013/01/02 12:08:12 UTC
[jira] [Updated] (HBASE-6811)
TestDrainingServer#testDrainingServerWithAbort sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HBASE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-6811:
---------------------------
Resolution: Fixed
Fix Version/s: 0.96.0
Status: Resolved (was: Patch Available)
> TestDrainingServer#testDrainingServerWithAbort sometimes fails in trunk
> -----------------------------------------------------------------------
>
> Key: HBASE-6811
> URL: https://issues.apache.org/jira/browse/HBASE-6811
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.96.0
> Reporter: Ted Yu
> Assignee: nkeywal
> Fix For: 0.96.0
>
> Attachments: 6811.v1.patch, 6811.v1.patch
>
>
> TestDrainingServer#testDrainingServerWithAbort failed in trunk build #3348:
> {code}
> Error Message
> Test conditions are not met: regions were created/deleted during the test. expected:<27> but was:<24>
> Stacktrace
> junit.framework.AssertionFailedError: Test conditions are not met: regions were created/deleted during the test. expected:<27> but was:<24>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:134)
> at org.apache.hadoop.hbase.TestDrainingServer.testDrainingServerWithAbort(TestDrainingServer.java:241)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
> Standard Output
> Shutting down the Mini HDFS Cluster
> Shutting down DataNode 4
> Shutting down DataNode 3
> Shutting down DataNode 2
> Shutting down DataNode 1
> Shutting down DataNode 0
> Standard Error
> 2012-09-18 20:18:30,026 INFO [pool-1-thread-1] hbase.ResourceChecker(144): before TestDrainingServer#testDrainingServerWithAbort: 441 threads, 700 file descriptors 7 connections,
> 2012-09-18 20:18:30,044 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer
> 2012-09-18 20:18:30,044 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1141): master:35050-0x139db080e690000 Retrieved 6 byte(s) of data from znode /hbase/balancer and set watcher; PBUF\x08\x00
> 2012-09-18 20:18:30,045 DEBUG [IPC Server handler 2 on 35050] zookeeper.ZKUtil(1141): master:35050-0x139db080e690000 Retrieved 6 byte(s) of data from znode /hbase/balancer and set watcher; PBUF\x08\x00
> 2012-09-18 20:18:30,045 INFO [IPC Server handler 2 on 35050] master.HMaster(1363): BalanceSwitch=false
> 2012-09-18 20:18:30,047 INFO [Thread-604] hbase.TestDrainingServer(211): Regions of drained server are: [t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.]
> 2012-09-18 20:18:30,047 INFO [Thread-604] hbase.TestDrainingServer(112): Making hemera.apache.org,33334,1347999502311 the draining server; it has 1 online regions
> 2012-09-18 20:18:30,048 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/draining
> 2012-09-18 20:18:30,049 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/draining/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,049 INFO [pool-1-thread-1-EventThread] zookeeper.DrainingServerTracker(83): Draining RS node created, adding to list [hemera.apache.org,33334,1347999502311]
> 2012-09-18 20:18:30,049 INFO [Thread-604] hbase.TestDrainingServer(220): The available servers are: [hemera.apache.org,38814,1347999502374, hemera.apache.org,58959,1347999502361, hemera.apache.org,43875,1347999502347, hemera.apache.org,51601,1347999502333]
> 2012-09-18 20:18:30,049 FATAL [Thread-604] regionserver.HRegionServer(1811): ABORTING region server hemera.apache.org,51601,1347999502333: Aborting
> 2012-09-18 20:18:30,050 FATAL [Thread-604] regionserver.HRegionServer(1818): RegionServer abort: loaded coprocessors are: []
> 2012-09-18 20:18:30,054 INFO [Thread-604] regionserver.HRegionServer(1821): Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=3, numberOfStores=3, numberOfStorefiles=0, storefileIndexSizeMB=0, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=167, maxHeapMB=1688, blockCacheSizeMB=1.98, blockCacheFreeMB=420.25, blockCacheCount=1, blockCacheHitCount=41, blockCacheMissCount=1, blockCacheEvictedCount=0, blockCacheHitRatio=97%, blockCacheHitCachingRatio=97%, hdfsBlocksLocalityIndex=0, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsPreadLatencyHistogramMean=0, fsPreadLatencyHistogramCount=0, fsPreadLatencyHistogramMedian=0, fsPreadLatencyHistogram75th=0, fsPreadLatencyHistogram95th=0, fsPreadLatencyHistogram99th=0, fsPreadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
> 2012-09-18 20:18:30,056 ERROR [IPC Server handler 0 on 35050] master.HMaster(1193): Region server ��hemera.apache.org,51601,1347999502333 reported a fatal error:
> ABORTING region server hemera.apache.org,51601,1347999502333: Aborting
> 2012-09-18 20:18:30,058 INFO [Thread-604] regionserver.HRegionServer(1737): STOPPED: Aborting
> 2012-09-18 20:18:30,058 FATAL [Thread-604] regionserver.HRegionServer(1811): ABORTING region server hemera.apache.org,43875,1347999502347: Aborting
> 2012-09-18 20:18:30,058 FATAL [Thread-604] regionserver.HRegionServer(1818): RegionServer abort: loaded coprocessors are: []
> 2012-09-18 20:18:30,062 INFO [Thread-604] regionserver.HRegionServer(1821): Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=7, numberOfStores=7, numberOfStorefiles=0, storefileIndexSizeMB=0, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=167, maxHeapMB=1688, blockCacheSizeMB=1.98, blockCacheFreeMB=420.25, blockCacheCount=1, blockCacheHitCount=41, blockCacheMissCount=1, blockCacheEvictedCount=0, blockCacheHitRatio=97%, blockCacheHitCachingRatio=97%, hdfsBlocksLocalityIndex=0, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsPreadLatencyHistogramMean=0, fsPreadLatencyHistogramCount=0, fsPreadLatencyHistogramMedian=0, fsPreadLatencyHistogram75th=0, fsPreadLatencyHistogram95th=0, fsPreadLatencyHistogram99th=0, fsPreadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
> 2012-09-18 20:18:30,062 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.SplitLogWorker(522): Sending interrupt to stop the worker thread
> 2012-09-18 20:18:30,063 INFO [SplitLogWorker-hemera.apache.org,51601,1347999502333] regionserver.SplitLogWorker(206): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
> 2012-09-18 20:18:30,063 INFO [SplitLogWorker-hemera.apache.org,51601,1347999502333] regionserver.SplitLogWorker(170): SplitLogWorker hemera.apache.org,51601,1347999502333 exiting
> 2012-09-18 20:18:30,063 ERROR [IPC Server handler 1 on 35050] master.HMaster(1193): Region server ��hemera.apache.org,43875,1347999502347 reported a fatal error:
> ABORTING region server hemera.apache.org,43875,1347999502347: Aborting
> 2012-09-18 20:18:30,063 INFO [Thread-604] regionserver.HRegionServer(1737): STOPPED: Aborting
> 2012-09-18 20:18:30,063 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.compactionChecker] hbase.Chore(81): RegionServer:1;hemera.apache.org,51601,1347999502333.compactionChecker exiting
> 2012-09-18 20:18:30,063 FATAL [Thread-604] regionserver.HRegionServer(1811): ABORTING region server hemera.apache.org,58959,1347999502361: Aborting
> 2012-09-18 20:18:30,063 DEBUG [pool-1-thread-1.LruBlockCache.EvictionThread] hfile.LruBlockCache(418): Block cache LRU eviction started; Attempting to free -408721.95 KB of total=1.98 MB
> 2012-09-18 20:18:30,063 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.cacheFlusher] regionserver.MemStoreFlusher(264): RegionServer:1;hemera.apache.org,51601,1347999502333.cacheFlusher exiting
> 2012-09-18 20:18:30,064 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-0] handler.CloseRegionHandler(124): Processing close of t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.
> 2012-09-18 20:18:30,063 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.logRoller] regionserver.LogRoller(118): LogRoller exiting.
> 2012-09-18 20:18:30,064 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-0] regionserver.HRegion(954): Closing t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.: disabling compactions & flushes
> 2012-09-18 20:18:30,064 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(943): aborting server hemera.apache.org,51601,1347999502333
> 2012-09-18 20:18:30,064 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.SplitLogWorker(522): Sending interrupt to stop the worker thread
> 2012-09-18 20:18:30,064 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-1] handler.CloseRegionHandler(124): Processing close of t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.
> 2012-09-18 20:18:30,065 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.cacheFlusher] regionserver.MemStoreFlusher(264): RegionServer:2;hemera.apache.org,43875,1347999502347.cacheFlusher exiting
> 2012-09-18 20:18:30,065 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.logRoller] regionserver.LogRoller(118): LogRoller exiting.
> 2012-09-18 20:18:30,065 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.compactionChecker] hbase.Chore(81): RegionServer:2;hemera.apache.org,43875,1347999502347.compactionChecker exiting
> 2012-09-18 20:18:30,063 FATAL [Thread-604] regionserver.HRegionServer(1818): RegionServer abort: loaded coprocessors are: []
> 2012-09-18 20:18:30,065 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(124): Processing close of t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.
> 2012-09-18 20:18:30,065 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-1] regionserver.HRegion(954): Closing t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.: disabling compactions & flushes
> 2012-09-18 20:18:30,067 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(954): Closing t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.: disabling compactions & flushes
> 2012-09-18 20:18:30,065 INFO [SplitLogWorker-hemera.apache.org,43875,1347999502347] regionserver.SplitLogWorker(206): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
> 2012-09-18 20:18:30,067 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(943): aborting server hemera.apache.org,43875,1347999502347
> 2012-09-18 20:18:30,067 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@11a59ce
> 2012-09-18 20:18:30,064 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@f7bf2d
> 2012-09-18 20:18:30,064 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-0] regionserver.HRegion(975): Updates disabled for region t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.
> 2012-09-18 20:18:30,064 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-2] handler.CloseRegionHandler(124): Processing close of t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.
> 2012-09-18 20:18:30,067 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e690009
> 2012-09-18 20:18:30,067 INFO [Thread-604] regionserver.HRegionServer(1821): Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=4, numberOfStores=4, numberOfStorefiles=0, storefileIndexSizeMB=0, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=168, maxHeapMB=1688, blockCacheSizeMB=1.98, blockCacheFreeMB=420.25, blockCacheCount=1, blockCacheHitCount=41, blockCacheMissCount=1, blockCacheEvictedCount=0, blockCacheHitRatio=97%, blockCacheHitCachingRatio=97%, hdfsBlocksLocalityIndex=0, slowHLogAppendCount=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsPreadLatencyHistogramMean=0, fsPreadLatencyHistogramCount=0, fsPreadLatencyHistogramMedian=0, fsPreadLatencyHistogram75th=0, fsPreadLatencyHistogram95th=0, fsPreadLatencyHistogram99th=0, fsPreadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
> 2012-09-18 20:18:30,067 INFO [SplitLogWorker-hemera.apache.org,43875,1347999502347] regionserver.SplitLogWorker(170): SplitLogWorker hemera.apache.org,43875,1347999502347 exiting
> 2012-09-18 20:18:30,067 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(975): Updates disabled for region t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.
> 2012-09-18 20:18:30,067 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(124): Processing close of t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.
> 2012-09-18 20:18:30,069 ERROR [IPC Server handler 3 on 35050] master.HMaster(1193): Region server ��hemera.apache.org,58959,1347999502361 reported a fatal error:
> ABORTING region server hemera.apache.org,58959,1347999502361: Aborting
> 2012-09-18 20:18:30,069 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(954): Closing t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.: disabling compactions & flushes
> 2012-09-18 20:18:30,067 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-1] regionserver.HRegion(975): Updates disabled for region t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.
> 2012-09-18 20:18:30,066 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] handler.CloseRegionHandler(124): Processing close of t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.
> 2012-09-18 20:18:30,070 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e690007
> 2012-09-18 20:18:30,070 INFO [Thread-604] regionserver.HRegionServer(1737): STOPPED: Aborting
> 2012-09-18 20:18:30,070 INFO [StoreCloserThread-t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,070 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(975): Updates disabled for region t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.
> 2012-09-18 20:18:30,070 INFO [StoreCloserThread-t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,071 INFO [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-1] regionserver.HRegion(1023): Closed t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.
> 2012-09-18 20:18:30,068 INFO [StoreCloserThread-t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,071 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-1] handler.CloseRegionHandler(168): Closed region t,iii,1347999506669.ae0257433f2941163b1d397f3d4f6efc.
> 2012-09-18 20:18:30,071 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(1023): Closed t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.
> 2012-09-18 20:18:30,068 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-2] regionserver.HRegion(954): Closing t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.: disabling compactions & flushes
> 2012-09-18 20:18:30,071 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(168): Closed region t,eee,1347999506656.3b9d9fdc58f2319931d38a906c8a1289.
> 2012-09-18 20:18:30,071 INFO [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-0] regionserver.HRegion(1023): Closed t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.
> 2012-09-18 20:18:30,071 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(1091): Waiting on 2 regions to close
> 2012-09-18 20:18:30,071 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(1095): {73c7f7079688e986da1850e53fb74f9d=t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.}
> 2012-09-18 20:18:30,071 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.SplitLogWorker(522): Sending interrupt to stop the worker thread
> 2012-09-18 20:18:30,070 INFO [Thread-604] hbase.TestDrainingServer(239): Regions of drained server are: [t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.]
> 2012-09-18 20:18:30,072 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.cacheFlusher] regionserver.MemStoreFlusher(264): RegionServer:3;hemera.apache.org,58959,1347999502361.cacheFlusher exiting
> 2012-09-18 20:18:30,070 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(954): Closing t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.: disabling compactions & flushes
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(124): Processing close of t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.
> 2012-09-18 20:18:30,070 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(1091): Waiting on 8 regions to close
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(954): Closing t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.: disabling compactions & flushes
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] handler.CloseRegionHandler(124): Processing close of t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(975): Updates disabled for region t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.
> 2012-09-18 20:18:30,073 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(954): Closing t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.: disabling compactions & flushes
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] handler.CloseRegionHandler(124): Processing close of t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.
> 2012-09-18 20:18:30,073 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,072 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(975): Updates disabled for region t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.
> 2012-09-18 20:18:30,073 INFO [StoreCloserThread-t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,072 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.compactionChecker] hbase.Chore(81): RegionServer:3;hemera.apache.org,58959,1347999502361.compactionChecker exiting
> 2012-09-18 20:18:30,073 INFO [StoreCloserThread-t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,073 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(1023): Closed t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.
> 2012-09-18 20:18:30,075 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(1023): Closed t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.
> 2012-09-18 20:18:30,075 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(168): Closed region t,ggg,1347999506664.b48b882c9992c9773e9bfda020d2137c.
> 2012-09-18 20:18:30,072 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.logRoller] regionserver.LogRoller(118): LogRoller exiting.
> 2012-09-18 20:18:30,072 INFO [SplitLogWorker-hemera.apache.org,58959,1347999502361] regionserver.SplitLogWorker(206): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
> 2012-09-18 20:18:30,071 INFO [StoreCloserThread-t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,076 INFO [SplitLogWorker-hemera.apache.org,58959,1347999502361] regionserver.SplitLogWorker(170): SplitLogWorker hemera.apache.org,58959,1347999502361 exiting
> 2012-09-18 20:18:30,076 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(1023): Closed t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.
> 2012-09-18 20:18:30,071 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-0] handler.CloseRegionHandler(168): Closed region t,lll,1347999506678.1c1644d5428c228bace562ef1bed5118.
> 2012-09-18 20:18:30,071 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(124): Processing close of t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.
> 2012-09-18 20:18:30,071 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-2] regionserver.HRegion(975): Updates disabled for region t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.
> 2012-09-18 20:18:30,079 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(168): Closed region t,kkk,1347999506675.d91c2dc99d3bac1d51686c9a6b4e74aa.
> 2012-09-18 20:18:30,079 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(124): Processing close of t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.
> 2012-09-18 20:18:30,078 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(954): Closing t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.: disabling compactions & flushes
> 2012-09-18 20:18:30,076 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(124): Processing close of t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.
> 2012-09-18 20:18:30,075 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] handler.CloseRegionHandler(168): Closed region t,fff,1347999506661.7d7428c069f7e46b3ded4161a6d8d961.
> 2012-09-18 20:18:30,079 INFO [StoreCloserThread-t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,073 INFO [pool-1-thread-1-EventThread] zookeeper.DrainingServerTracker(101): Draining RS node deleted, removing from list [hemera.apache.org,33334,1347999502311]
> 2012-09-18 20:18:30,084 INFO [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-2] regionserver.HRegion(1023): Closed t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.
> 2012-09-18 20:18:30,084 INFO [pool-1-thread-1] hbase.ResourceChecker(144): after TestDrainingServer#testDrainingServerWithAbort: 348 threads (was 441), 584 file descriptors (was 700). 5 connections (was 7),
> 2012-09-18 20:18:30,073 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(954): Closing t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.: disabling compactions & flushes
> 2012-09-18 20:18:30,073 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(975): Updates disabled for region t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.
> 2012-09-18 20:18:30,072 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(943): aborting server hemera.apache.org,58959,1347999502361
> 2012-09-18 20:18:30,086 INFO [StoreCloserThread-t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,072 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(1095): {c1e5810326004add2ab532ccc9e8c24e=t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e., 6b45157a30794a82fc5cbdb3589032e2=t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2., 95255bd4b3285804a2c818d1bf7f459f=t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f., cb765c4343d7b8c707c88b4d4b79a165=t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165., 78622cfb2af5eca037e2375d7566cac6=t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.}
> 2012-09-18 20:18:30,086 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(1023): Closed t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.
> 2012-09-18 20:18:30,086 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] handler.CloseRegionHandler(168): Closed region t,hhh,1347999506667.5a1faf1d76da0300d16f9c8dbb752824.
> 2012-09-18 20:18:30,086 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] handler.CloseRegionHandler(124): Processing close of t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.
> 2012-09-18 20:18:30,086 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1880b02
> 2012-09-18 20:18:30,086 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(975): Updates disabled for region t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.
> 2012-09-18 20:18:30,084 DEBUG [RS_CLOSE_REGION-hemera.apache.org,51601,1347999502333-2] handler.CloseRegionHandler(168): Closed region t,ttt,1347999506703.73c7f7079688e986da1850e53fb74f9d.
> 2012-09-18 20:18:30,084 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/draining
> 2012-09-18 20:18:30,084 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(954): Closing t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.: disabling compactions & flushes
> 2012-09-18 20:18:30,081 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] handler.CloseRegionHandler(124): Processing close of t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.
> 2012-09-18 20:18:30,090 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(954): Closing t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.: disabling compactions & flushes
> 2012-09-18 20:18:30,090 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(975): Updates disabled for region t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.
> 2012-09-18 20:18:30,079 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(975): Updates disabled for region t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.
> 2012-09-18 20:18:30,079 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(954): Closing t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.: disabling compactions & flushes
> 2012-09-18 20:18:30,090 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(975): Updates disabled for region t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.
> 2012-09-18 20:18:30,090 INFO [StoreCloserThread-t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,090 INFO [StoreCloserThread-t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,089 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e690008
> 2012-09-18 20:18:30,091 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(1023): Closed t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.
> 2012-09-18 20:18:30,091 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] handler.CloseRegionHandler(168): Closed region t,www,1347999506711.985b3c0de2cbfe789c69850bb128887d.
> 2012-09-18 20:18:30,091 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] handler.CloseRegionHandler(124): Processing close of t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.
> 2012-09-18 20:18:30,089 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(954): Closing t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.: disabling compactions & flushes
> 2012-09-18 20:18:30,091 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(975): Updates disabled for region t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.
> 2012-09-18 20:18:30,091 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(954): Closing t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.: disabling compactions & flushes
> 2012-09-18 20:18:30,091 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(975): Updates disabled for region t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.
> 2012-09-18 20:18:30,091 INFO [StoreCloserThread-t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,092 INFO [StoreCloserThread-t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,092 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(1023): Closed t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.
> 2012-09-18 20:18:30,092 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] regionserver.HRegion(1023): Closed t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.
> 2012-09-18 20:18:30,092 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(168): Closed region t,nnn,1347999506684.a2e6ab866441d82e1124ad6dcaf7fa03.
> 2012-09-18 20:18:30,091 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(1023): Closed t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.
> 2012-09-18 20:18:30,092 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(168): Closed region t,sss,1347999506698.c1e5810326004add2ab532ccc9e8c24e.
> 2012-09-18 20:18:30,092 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(747): Shutting down minicluster
> 2012-09-18 20:18:30,092 DEBUG [pool-1-thread-1] util.JVMClusterUtil(223): Shutting down HBase Cluster
> 2012-09-18 20:18:30,090 INFO [StoreCloserThread-t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,090 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(975): Updates disabled for region t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.
> 2012-09-18 20:18:30,093 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] regionserver.HRegion(1023): Closed t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.
> 2012-09-18 20:18:30,092 INFO [pool-1-thread-1] master.HMaster(2049): Cluster shutdown requested
> 2012-09-18 20:18:30,092 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(124): Processing close of t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.
> 2012-09-18 20:18:30,093 INFO [StoreCloserThread-t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,092 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-1] handler.CloseRegionHandler(168): Closed region t,rrr,1347999506695.29ca9f1226c85d9ad4523735386186da.
> 2012-09-18 20:18:30,093 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(1023): Closed t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.
> 2012-09-18 20:18:30,092 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.CompactSplitThread(253): Waiting for Split Thread to finish...
> 2012-09-18 20:18:30,093 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.CompactSplitThread(253): Waiting for Large Compaction Thread to finish...
> 2012-09-18 20:18:30,091 INFO [StoreCloserThread-t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,093 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.CompactSplitThread(253): Waiting for Small Compaction Thread to finish...
> 2012-09-18 20:18:30,094 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311-EventThread] zookeeper.ZooKeeperWatcher(265): regionserver:33334-0x139db080e690002 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,093 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(168): Closed region t,,1347999506624.6b45157a30794a82fc5cbdb3589032e2.
> 2012-09-18 20:18:30,093 INFO [hemera.apache.org,35050,1347999502111-CatalogJanitor] hbase.Chore(81): hemera.apache.org,35050,1347999502111-CatalogJanitor exiting
> 2012-09-18 20:18:30,093 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(954): Closing t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.: disabling compactions & flushes
> 2012-09-18 20:18:30,093 INFO [Master:0;hemera.apache.org,35050,1347999502111] master.ServerManager(398): Waiting on regionserver(s) to go down hemera.apache.org,38814,1347999502374, hemera.apache.org,58959,1347999502361, hemera.apache.org,43875,1347999502347, hemera.apache.org,33334,1347999502311, hemera.apache.org,51601,1347999502333
> 2012-09-18 20:18:30,093 INFO [hemera.apache.org,35050,1347999502111-BalancerChore] hbase.Chore(81): hemera.apache.org,35050,1347999502111-BalancerChore exiting
> 2012-09-18 20:18:30,093 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-1] handler.CloseRegionHandler(168): Closed region t,xxx,1347999506714.95255bd4b3285804a2c818d1bf7f459f.
> 2012-09-18 20:18:30,094 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(975): Updates disabled for region t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.
> 2012-09-18 20:18:30,094 DEBUG [RS_CLOSE_ROOT-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(124): Processing close of -ROOT-,,0.70236052
> 2012-09-18 20:18:30,094 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(1091): Waiting on 2 regions to close
> 2012-09-18 20:18:30,094 INFO [StoreCloserThread-t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,094 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(124): Processing close of t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.
> 2012-09-18 20:18:30,095 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] regionserver.HRegion(1023): Closed t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.
> 2012-09-18 20:18:30,094 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361-EventThread] zookeeper.ZooKeeperWatcher(265): regionserver:58959-0x139db080e690005 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,094 INFO [pool-1-thread-1] regionserver.HRegionServer(1737): STOPPED: Shutdown requested
> 2012-09-18 20:18:30,094 INFO [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] regionserver.HRegion(1023): Closed t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.
> 2012-09-18 20:18:30,094 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333-EventThread] zookeeper.ZooKeeperWatcher(265): regionserver:51601-0x139db080e690001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,093 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374-EventThread] zookeeper.ZooKeeperWatcher(265): regionserver:38814-0x139db080e690003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,093 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347-EventThread] zookeeper.ZooKeeperWatcher(265): regionserver:43875-0x139db080e690004 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,093 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/shutdown
> 2012-09-18 20:18:30,095 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361-EventThread] zookeeper.ZKUtil(237): regionserver:58959-0x139db080e690005 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,095 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374-EventThread] zookeeper.ZKUtil(237): regionserver:38814-0x139db080e690003 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,095 DEBUG [RS_CLOSE_REGION-hemera.apache.org,58959,1347999502361-2] handler.CloseRegionHandler(168): Closed region t,ooo,1347999506686.c4178dd0ed59cff7c16b018f8936b194.
> 2012-09-18 20:18:30,095 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-0] handler.CloseRegionHandler(168): Closed region t,qqq,1347999506692.cb765c4343d7b8c707c88b4d4b79a165.
> 2012-09-18 20:18:30,096 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(237): master:35050-0x139db080e690000 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,095 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(954): Closing t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.: disabling compactions & flushes
> 2012-09-18 20:18:30,094 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(1095): {70236052=-ROOT-,,0.70236052}
> 2012-09-18 20:18:30,094 DEBUG [RS_CLOSE_ROOT-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(954): Closing -ROOT-,,0.70236052: disabling compactions & flushes
> 2012-09-18 20:18:30,094 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311-EventThread] zookeeper.ZKUtil(237): regionserver:33334-0x139db080e690002 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,096 DEBUG [RS_CLOSE_ROOT-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(975): Updates disabled for region -ROOT-,,0.70236052
> 2012-09-18 20:18:30,096 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(975): Updates disabled for region t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.
> 2012-09-18 20:18:30,096 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347-EventThread] zookeeper.ZKUtil(237): regionserver:43875-0x139db080e690004 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,095 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.SplitLogWorker(522): Sending interrupt to stop the worker thread
> 2012-09-18 20:18:30,096 INFO [StoreCloserThread-t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,098 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.cacheFlusher] regionserver.MemStoreFlusher(264): RegionServer:0;hemera.apache.org,33334,1347999502311.cacheFlusher exiting
> 2012-09-18 20:18:30,098 INFO [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] regionserver.HRegion(1023): Closed t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.
> 2012-09-18 20:18:30,098 INFO [StoreCloserThread--ROOT-,,0.70236052-1] regionserver.HStore(635): Closed info
> 2012-09-18 20:18:30,095 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333-EventThread] zookeeper.ZKUtil(237): regionserver:51601-0x139db080e690001 /hbase/shutdown does not exist. Watcher is set.
> 2012-09-18 20:18:30,098 INFO [RS_CLOSE_ROOT-hemera.apache.org,58959,1347999502361-0] regionserver.HRegion(1023): Closed -ROOT-,,0.70236052
> 2012-09-18 20:18:30,098 DEBUG [RS_CLOSE_REGION-hemera.apache.org,43875,1347999502347-2] handler.CloseRegionHandler(168): Closed region t,mmm,1347999506681.78622cfb2af5eca037e2375d7566cac6.
> 2012-09-18 20:18:30,098 DEBUG [RS_CLOSE_REGION-hemera.apache.org,33334,1347999502311-0] handler.CloseRegionHandler(124): Processing close of t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.
> 2012-09-18 20:18:30,098 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.compactionChecker] hbase.Chore(81): RegionServer:0;hemera.apache.org,33334,1347999502311.compactionChecker exiting
> 2012-09-18 20:18:30,098 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.logRoller] regionserver.LogRoller(118): LogRoller exiting.
> 2012-09-18 20:18:30,098 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.HRegionServer(947): stopping server hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,098 INFO [SplitLogWorker-hemera.apache.org,33334,1347999502311] regionserver.SplitLogWorker(206): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
> 2012-09-18 20:18:30,098 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5afcb1
> 2012-09-18 20:18:30,098 DEBUG [RS_CLOSE_REGION-hemera.apache.org,33334,1347999502311-0] regionserver.HRegion(954): Closing t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.: disabling compactions & flushes
> 2012-09-18 20:18:30,098 DEBUG [RS_CLOSE_ROOT-hemera.apache.org,58959,1347999502361-0] handler.CloseRegionHandler(168): Closed region -ROOT-,,0.70236052
> 2012-09-18 20:18:30,099 DEBUG [RS_CLOSE_REGION-hemera.apache.org,33334,1347999502311-0] regionserver.HRegion(975): Updates disabled for region t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.
> 2012-09-18 20:18:30,099 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e69000b
> 2012-09-18 20:18:30,099 INFO [SplitLogWorker-hemera.apache.org,33334,1347999502311] regionserver.SplitLogWorker(170): SplitLogWorker hemera.apache.org,33334,1347999502311 exiting
> 2012-09-18 20:18:30,099 INFO [StoreCloserThread-t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,099 INFO [RS_CLOSE_REGION-hemera.apache.org,33334,1347999502311-0] regionserver.HRegion(1023): Closed t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.
> 2012-09-18 20:18:30,100 DEBUG [RS_CLOSE_REGION-hemera.apache.org,33334,1347999502311-0] handler.CloseRegionHandler(168): Closed region t,yyy,1347999506717.bdd1a3ab3105de63248038bd0eb40564.
> 2012-09-18 20:18:30,100 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.HRegionServer(966): stopping server hemera.apache.org,33334,1347999502311; all regions closed.
> 2012-09-18 20:18:30,100 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.logSyncer] wal.HLog$LogSyncer(1245): RegionServer:0;hemera.apache.org,33334,1347999502311.logSyncer exiting
> 2012-09-18 20:18:30,100 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] wal.HLog(1007): closing hlog writer in hdfs://localhost:52077/user/jenkins/hbase/.logs/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,272 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(966): stopping server hemera.apache.org,51601,1347999502333; all regions closed.
> 2012-09-18 20:18:30,272 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.logSyncer] wal.HLog$LogSyncer(1245): RegionServer:1;hemera.apache.org,51601,1347999502333.logSyncer exiting
> 2012-09-18 20:18:30,272 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] wal.HLog(1007): closing hlog writer in hdfs://localhost:52077/user/jenkins/hbase/.logs/hemera.apache.org,51601,1347999502333
> 2012-09-18 20:18:30,284 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.leaseChecker] regionserver.Leases(123): RegionServer:0;hemera.apache.org,33334,1347999502311.leaseChecker closing leases
> 2012-09-18 20:18:30,284 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311.leaseChecker] regionserver.Leases(130): RegionServer:0;hemera.apache.org,33334,1347999502311.leaseChecker closed leases
> 2012-09-18 20:18:30,285 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.leaseChecker] regionserver.Leases(123): RegionServer:1;hemera.apache.org,51601,1347999502333.leaseChecker closing leases
> 2012-09-18 20:18:30,285 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.leaseChecker] regionserver.Leases(123): RegionServer:2;hemera.apache.org,43875,1347999502347.leaseChecker closing leases
> 2012-09-18 20:18:30,285 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.leaseChecker] regionserver.Leases(130): RegionServer:2;hemera.apache.org,43875,1347999502347.leaseChecker closed leases
> 2012-09-18 20:18:30,286 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.Leases(123): RegionServer:1;hemera.apache.org,51601,1347999502333 closing leases
> 2012-09-18 20:18:30,286 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.Leases(130): RegionServer:1;hemera.apache.org,51601,1347999502333 closed leases
> 2012-09-18 20:18:30,285 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333.leaseChecker] regionserver.Leases(130): RegionServer:1;hemera.apache.org,51601,1347999502333.leaseChecker closed leases
> 2012-09-18 20:18:30,286 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.CompactSplitThread(253): Waiting for Split Thread to finish...
> 2012-09-18 20:18:30,286 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(966): stopping server hemera.apache.org,43875,1347999502347; all regions closed.
> 2012-09-18 20:18:30,286 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.CompactSplitThread(253): Waiting for Large Compaction Thread to finish...
> 2012-09-18 20:18:30,287 DEBUG [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.CompactSplitThread(253): Waiting for Small Compaction Thread to finish...
> 2012-09-18 20:18:30,287 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347.logSyncer] wal.HLog$LogSyncer(1245): RegionServer:2;hemera.apache.org,43875,1347999502347.logSyncer exiting
> 2012-09-18 20:18:30,287 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] wal.HLog(1007): closing hlog writer in hdfs://localhost:52077/user/jenkins/hbase/.logs/hemera.apache.org,43875,1347999502347
> 2012-09-18 20:18:30,287 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.leaseChecker] regionserver.Leases(123): RegionServer:3;hemera.apache.org,58959,1347999502361.leaseChecker closing leases
> 2012-09-18 20:18:30,288 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.leaseChecker] regionserver.Leases(130): RegionServer:3;hemera.apache.org,58959,1347999502361.leaseChecker closed leases
> 2012-09-18 20:18:30,289 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/hemera.apache.org,51601,1347999502333
> 2012-09-18 20:18:30,289 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [hemera.apache.org,51601,1347999502333]
> 2012-09-18 20:18:30,289 INFO [pool-1-thread-1-EventThread] master.ServerManager(446): Cluster shutdown set; hemera.apache.org,51601,1347999502333 expired; onlineServers=4
> 2012-09-18 20:18:30,289 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
> 2012-09-18 20:18:30,290 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(995): stopping server hemera.apache.org,51601,1347999502333; zookeeper connection closed.
> 2012-09-18 20:18:30,290 INFO [RegionServer:1;hemera.apache.org,51601,1347999502333] regionserver.HRegionServer(998): RegionServer:1;hemera.apache.org,51601,1347999502333 exiting
> 2012-09-18 20:18:30,290 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@21e115] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@21e115
> 2012-09-18 20:18:30,291 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:30,291 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,43875,1347999502347
> 2012-09-18 20:18:30,292 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,58959,1347999502361
> 2012-09-18 20:18:30,293 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,296 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(966): stopping server hemera.apache.org,58959,1347999502361; all regions closed.
> 2012-09-18 20:18:30,296 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361.logSyncer] wal.HLog$LogSyncer(1245): RegionServer:3;hemera.apache.org,58959,1347999502361.logSyncer exiting
> 2012-09-18 20:18:30,297 DEBUG [RegionServer:3;hemera.apache.org,58959,1347999502361] wal.HLog(1007): closing hlog writer in hdfs://localhost:52077/user/jenkins/hbase/.logs/hemera.apache.org,58959,1347999502361
> 2012-09-18 20:18:30,298 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.Leases(123): RegionServer:2;hemera.apache.org,43875,1347999502347 closing leases
> 2012-09-18 20:18:30,298 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.Leases(130): RegionServer:2;hemera.apache.org,43875,1347999502347 closed leases
> 2012-09-18 20:18:30,298 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.CompactSplitThread(253): Waiting for Split Thread to finish...
> 2012-09-18 20:18:30,298 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.CompactSplitThread(253): Waiting for Large Compaction Thread to finish...
> 2012-09-18 20:18:30,298 DEBUG [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.CompactSplitThread(253): Waiting for Small Compaction Thread to finish...
> 2012-09-18 20:18:30,300 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/hemera.apache.org,43875,1347999502347
> 2012-09-18 20:18:30,300 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [hemera.apache.org,43875,1347999502347]
> 2012-09-18 20:18:30,300 INFO [pool-1-thread-1-EventThread] master.ServerManager(446): Cluster shutdown set; hemera.apache.org,43875,1347999502347 expired; onlineServers=3
> 2012-09-18 20:18:30,301 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
> 2012-09-18 20:18:30,301 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(995): stopping server hemera.apache.org,43875,1347999502347; zookeeper connection closed.
> 2012-09-18 20:18:30,301 INFO [RegionServer:2;hemera.apache.org,43875,1347999502347] regionserver.HRegionServer(998): RegionServer:2;hemera.apache.org,43875,1347999502347 exiting
> 2012-09-18 20:18:30,302 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@147e54e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@147e54e
> 2012-09-18 20:18:30,302 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:30,303 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,58959,1347999502361
> 2012-09-18 20:18:30,303 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,307 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.Leases(123): RegionServer:3;hemera.apache.org,58959,1347999502361 closing leases
> 2012-09-18 20:18:30,308 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.Leases(130): RegionServer:3;hemera.apache.org,58959,1347999502361 closed leases
> 2012-09-18 20:18:30,309 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/hemera.apache.org,58959,1347999502361
> 2012-09-18 20:18:30,309 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [hemera.apache.org,58959,1347999502361]
> 2012-09-18 20:18:30,309 INFO [pool-1-thread-1-EventThread] master.ServerManager(446): Cluster shutdown set; hemera.apache.org,58959,1347999502361 expired; onlineServers=2
> 2012-09-18 20:18:30,309 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
> 2012-09-18 20:18:30,310 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(995): stopping server hemera.apache.org,58959,1347999502361; zookeeper connection closed.
> 2012-09-18 20:18:30,310 INFO [RegionServer:3;hemera.apache.org,58959,1347999502361] regionserver.HRegionServer(998): RegionServer:3;hemera.apache.org,58959,1347999502361 exiting
> 2012-09-18 20:18:30,310 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1c7510d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1c7510d
> 2012-09-18 20:18:30,310 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:30,311 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(869): Closing user regions
> 2012-09-18 20:18:30,311 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,311 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] handler.CloseRegionHandler(124): Processing close of t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.
> 2012-09-18 20:18:30,311 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(954): Closing t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.: disabling compactions & flushes
> 2012-09-18 20:18:30,311 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(124): Processing close of t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.
> 2012-09-18 20:18:30,311 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(975): Updates disabled for region t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.
> 2012-09-18 20:18:30,312 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(954): Closing t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.: disabling compactions & flushes
> 2012-09-18 20:18:30,312 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(975): Updates disabled for region t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.
> 2012-09-18 20:18:30,312 INFO [StoreCloserThread-t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,311 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] handler.CloseRegionHandler(124): Processing close of t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.
> 2012-09-18 20:18:30,312 INFO [StoreCloserThread-t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,312 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(954): Closing t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.: disabling compactions & flushes
> 2012-09-18 20:18:30,313 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1023): Closed t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.
> 2012-09-18 20:18:30,313 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(975): Updates disabled for region t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.
> 2012-09-18 20:18:30,313 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(168): Closed region t,vvv,1347999506708.df029f1344d578333d45fbd32dbbba23.
> 2012-09-18 20:18:30,312 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(1023): Closed t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.
> 2012-09-18 20:18:30,313 INFO [StoreCloserThread-t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,313 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(124): Processing close of t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.
> 2012-09-18 20:18:30,313 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(1023): Closed t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.
> 2012-09-18 20:18:30,313 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(954): Closing t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.: disabling compactions & flushes
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] handler.CloseRegionHandler(168): Closed region t,ppp,1347999506690.815416a53df0b3b68fc865a0edf53203.
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(975): Updates disabled for region t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] handler.CloseRegionHandler(124): Processing close of t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.
> 2012-09-18 20:18:30,313 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] handler.CloseRegionHandler(168): Closed region t,bbb,1347999506644.8a6358bbd5596deaad1f278da7ca32b4.
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(954): Closing t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.: disabling compactions & flushes
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] handler.CloseRegionHandler(124): Processing close of t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.
> 2012-09-18 20:18:30,314 INFO [StoreCloserThread-t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(954): Closing t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.: disabling compactions & flushes
> 2012-09-18 20:18:30,315 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(975): Updates disabled for region t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.
> 2012-09-18 20:18:30,314 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(975): Updates disabled for region t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.
> 2012-09-18 20:18:30,315 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1023): Closed t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.
> 2012-09-18 20:18:30,315 INFO [StoreCloserThread-t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,315 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(168): Closed region t,jjj,1347999506672.393b3166f0a447101593a6e538c0a2fd.
> 2012-09-18 20:18:30,315 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] regionserver.HRegion(1023): Closed t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.
> 2012-09-18 20:18:30,316 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-1] handler.CloseRegionHandler(168): Closed region t,ccc,1347999506650.8568717cae604e1dab3b78bfeff881e9.
> 2012-09-18 20:18:30,315 INFO [StoreCloserThread-t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,316 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(124): Processing close of t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.
> 2012-09-18 20:18:30,316 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] regionserver.HRegion(1023): Closed t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.
> 2012-09-18 20:18:30,316 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(954): Closing t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.: disabling compactions & flushes
> 2012-09-18 20:18:30,316 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-2] handler.CloseRegionHandler(168): Closed region t,ddd,1347999506653.d94a101bf8bc56ad5a83b8b626f8189b.
> 2012-09-18 20:18:30,316 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(975): Updates disabled for region t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.
> 2012-09-18 20:18:30,317 INFO [StoreCloserThread-t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.-1] regionserver.HStore(635): Closed f
> 2012-09-18 20:18:30,317 INFO [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1023): Closed t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.
> 2012-09-18 20:18:30,317 DEBUG [RS_CLOSE_REGION-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(168): Closed region t,uuu,1347999506706.bf4ceeff334c5a35fa3a54272c2f395f.
> 2012-09-18 20:18:30,392 INFO [hemera.apache.org,35050,1347999502111.splitLogManagerTimeoutMonitor] hbase.Chore(81): hemera.apache.org,35050,1347999502111.splitLogManagerTimeoutMonitor exiting
> 2012-09-18 20:18:30,507 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] wal.HLog(975): Moved 1 log files to /user/jenkins/hbase/.oldlogs
> 2012-09-18 20:18:30,509 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.Leases(123): RegionServer:0;hemera.apache.org,33334,1347999502311 closing leases
> 2012-09-18 20:18:30,509 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.Leases(130): RegionServer:0;hemera.apache.org,33334,1347999502311 closed leases
> 2012-09-18 20:18:30,509 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.CompactSplitThread(253): Waiting for Split Thread to finish...
> 2012-09-18 20:18:30,509 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.CompactSplitThread(253): Waiting for Large Compaction Thread to finish...
> 2012-09-18 20:18:30,509 DEBUG [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.CompactSplitThread(253): Waiting for Small Compaction Thread to finish...
> 2012-09-18 20:18:30,510 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/hemera.apache.org,33334,1347999502311
> 2012-09-18 20:18:30,511 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [hemera.apache.org,33334,1347999502311]
> 2012-09-18 20:18:30,511 INFO [pool-1-thread-1-EventThread] master.ServerManager(446): Cluster shutdown set; hemera.apache.org,33334,1347999502311 expired; onlineServers=1
> 2012-09-18 20:18:30,511 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
> 2012-09-18 20:18:30,512 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.HRegionServer(995): stopping server hemera.apache.org,33334,1347999502311; zookeeper connection closed.
> 2012-09-18 20:18:30,512 INFO [RegionServer:0;hemera.apache.org,33334,1347999502311] regionserver.HRegionServer(998): RegionServer:0;hemera.apache.org,33334,1347999502311 exiting
> 2012-09-18 20:18:30,512 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1ee9cc3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1ee9cc3
> 2012-09-18 20:18:30,513 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(235): master:35050-0x139db080e690000 Set watcher on existing znode /hbase/rs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:30,513 INFO [pool-1-thread-1] regionserver.HRegionServer(1737): STOPPED: Shutdown requested
> 2012-09-18 20:18:30,514 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.SplitLogWorker(522): Sending interrupt to stop the worker thread
> 2012-09-18 20:18:30,514 INFO [SplitLogWorker-hemera.apache.org,38814,1347999502374] regionserver.SplitLogWorker(206): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
> 2012-09-18 20:18:30,514 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.compactionChecker] hbase.Chore(81): RegionServer:4;hemera.apache.org,38814,1347999502374.compactionChecker exiting
> 2012-09-18 20:18:30,514 INFO [SplitLogWorker-hemera.apache.org,38814,1347999502374] regionserver.SplitLogWorker(170): SplitLogWorker hemera.apache.org,38814,1347999502374 exiting
> 2012-09-18 20:18:30,514 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(947): stopping server hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:30,514 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.logRoller] regionserver.LogRoller(118): LogRoller exiting.
> 2012-09-18 20:18:30,514 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.cacheFlusher] regionserver.MemStoreFlusher(264): RegionServer:4;hemera.apache.org,38814,1347999502374.cacheFlusher exiting
> 2012-09-18 20:18:30,514 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@df39bc
> 2012-09-18 20:18:30,515 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e69000a
> 2012-09-18 20:18:30,516 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.CompactSplitThread(253): Waiting for Split Thread to finish...
> 2012-09-18 20:18:30,516 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.CompactSplitThread(253): Waiting for Large Compaction Thread to finish...
> 2012-09-18 20:18:30,516 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.CompactSplitThread(253): Waiting for Small Compaction Thread to finish...
> 2012-09-18 20:18:30,517 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(1091): Waiting on 1 regions to close
> 2012-09-18 20:18:30,517 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(124): Processing close of .META.,,1.1028785192
> 2012-09-18 20:18:30,517 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(1095): {1028785192=.META.,,1.1028785192}
> 2012-09-18 20:18:30,517 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(954): Closing .META.,,1.1028785192: disabling compactions & flushes
> 2012-09-18 20:18:30,518 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(975): Updates disabled for region .META.,,1.1028785192
> 2012-09-18 20:18:30,518 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1481): Started memstore flush for .META.,,1.1028785192, current region memstore size 20.1k
> 2012-09-18 20:18:30,518 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1528): Finished snapshotting .META.,,1.1028785192, commencing wait for mvcc, flushsize=20560
> 2012-09-18 20:18:30,518 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1538): Finished snapshotting, commencing flushing stores
> 2012-09-18 20:18:30,521 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] util.FSUtils(167): Creating file=hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/.tmp/6c449086188648c08a990537dba22ca4 with permission=rwxrwxrwx
> 2012-09-18 20:18:30,523 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] hfile.HFileWriterV2(142): Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
> 2012-09-18 20:18:30,524 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.StoreFile$Writer(1021): Delete Family Bloom filter type for hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/.tmp/6c449086188648c08a990537dba22ca4: CompoundBloomFilterWriter
> 2012-09-18 20:18:30,939 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.StoreFile$Writer(1241): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/.tmp/6c449086188648c08a990537dba22ca4)
> 2012-09-18 20:18:30,939 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HStore(767): Flushed , sequenceid=59, memsize=20.1k, into tmp file hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/.tmp/6c449086188648c08a990537dba22ca4
> 2012-09-18 20:18:30,948 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HStore(792): Renaming flushed file at hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/.tmp/6c449086188648c08a990537dba22ca4 to hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/info/6c449086188648c08a990537dba22ca4
> 2012-09-18 20:18:30,954 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HStore(815): Added hdfs://localhost:52077/user/jenkins/hbase/.META./1028785192/info/6c449086188648c08a990537dba22ca4, entries=89, sequenceid=59, filesize=10.2k
> 2012-09-18 20:18:30,955 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1617): Finished memstore flush of ~20.1k/20560, currentsize=0.0/0 for region .META.,,1.1028785192 in 437ms, sequenceid=59, compaction requested=false
> 2012-09-18 20:18:30,956 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(635): Closed info
> 2012-09-18 20:18:30,956 INFO [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] regionserver.HRegion(1023): Closed .META.,,1.1028785192
> 2012-09-18 20:18:30,956 DEBUG [RS_CLOSE_META-hemera.apache.org,38814,1347999502374-0] handler.CloseRegionHandler(168): Closed region .META.,,1.1028785192
> 2012-09-18 20:18:31,112 INFO [Master:0;hemera.apache.org,35050,1347999502111] master.ServerManager(398): Waiting on regionserver(s) to go down hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:31,118 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(966): stopping server hemera.apache.org,38814,1347999502374; all regions closed.
> 2012-09-18 20:18:31,118 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.logSyncer] wal.HLog$LogSyncer(1245): RegionServer:4;hemera.apache.org,38814,1347999502374.logSyncer exiting
> 2012-09-18 20:18:31,118 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] wal.HLog(1007): closing hlog writer in hdfs://localhost:52077/user/jenkins/hbase/.logs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:31,124 DEBUG [RegionServer:4;hemera.apache.org,38814,1347999502374] wal.HLog(975): Moved 1 log files to /user/jenkins/hbase/.oldlogs
> 2012-09-18 20:18:31,125 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.Leases(123): RegionServer:4;hemera.apache.org,38814,1347999502374 closing leases
> 2012-09-18 20:18:31,125 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.Leases(130): RegionServer:4;hemera.apache.org,38814,1347999502374 closed leases
> 2012-09-18 20:18:31,126 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/hemera.apache.org,38814,1347999502374
> 2012-09-18 20:18:31,126 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [hemera.apache.org,38814,1347999502374]
> 2012-09-18 20:18:31,126 INFO [pool-1-thread-1-EventThread] master.ServerManager(446): Cluster shutdown set; hemera.apache.org,38814,1347999502374 expired; onlineServers=0
> 2012-09-18 20:18:31,126 DEBUG [Master:0;hemera.apache.org,35050,1347999502111] master.HMaster(1067): Stopping service threads
> 2012-09-18 20:18:31,126 INFO [pool-1-thread-1-EventThread] master.HMaster(2049): Cluster shutdown set; onlineServer=0
> 2012-09-18 20:18:31,127 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(995): stopping server hemera.apache.org,38814,1347999502374; zookeeper connection closed.
> 2012-09-18 20:18:31,127 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374] regionserver.HRegionServer(998): RegionServer:4;hemera.apache.org,38814,1347999502374 exiting
> 2012-09-18 20:18:31,127 INFO [Master:0;hemera.apache.org,35050,1347999502111.oldLogCleaner] hbase.Chore(81): Master:0;hemera.apache.org,35050,1347999502111.oldLogCleaner exiting
> 2012-09-18 20:18:31,127 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): master:35050-0x139db080e690000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
> 2012-09-18 20:18:31,127 INFO [Master:0;hemera.apache.org,35050,1347999502111.archivedHFileCleaner] hbase.Chore(81): Master:0;hemera.apache.org,35050,1347999502111.archivedHFileCleaner exiting
> 2012-09-18 20:18:31,128 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@194f467] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@194f467
> 2012-09-18 20:18:31,128 INFO [pool-1-thread-1] util.JVMClusterUtil(263): Shutdown of 1 master(s) and 5 regionserver(s) complete
> 2012-09-18 20:18:31,128 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1523): Closing zookeeper sessionid=0x139db080e69000d
> 2012-09-18 20:18:31,129 DEBUG [Master:0;hemera.apache.org,35050,1347999502111] zookeeper.ZKUtil(1141): master:35050-0x139db080e690000 Retrieved 36 byte(s) of data from znode /hbase/master; data=PBUF\x0A\x1E\x0A\x11hemera.ap...
> 2012-09-18 20:18:31,130 DEBUG [Master:0;hemera.apache.org,35050,1347999502111] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1152c0e
> 2012-09-18 20:18:31,130 INFO [hemera.apache.org,35050,1347999502111.timeoutMonitor] hbase.Chore(81): hemera.apache.org,35050,1347999502111.timeoutMonitor exiting
> 2012-09-18 20:18:31,130 INFO [hemera.apache.org,35050,1347999502111.timerUpdater] hbase.Chore(81): hemera.apache.org,35050,1347999502111.timerUpdater exiting
> 2012-09-18 20:18:31,130 INFO [Master:0;hemera.apache.org,35050,1347999502111] master.HMaster(481): HMaster main thread exiting
> 2012-09-18 20:18:31,132 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(238): Shutdown MiniZK cluster with all ZK servers
> 2012-09-18 20:18:31,132 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:31,231 DEBUG [Master:0;hemera.apache.org,35050,1347999502111-EventThread] zookeeper.ZooKeeperWatcher(265): hconnection 0x1c1eceb-0x139db080e690006 Received ZooKeeper Event, type=None, state=Disconnected, path=null
> 2012-09-18 20:18:31,231 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(265): unittest-0x139db080e69000e Received ZooKeeper Event, type=None, state=Disconnected, path=null
> 2012-09-18 20:18:31,231 DEBUG [Master:0;hemera.apache.org,35050,1347999502111-EventThread] zookeeper.ZooKeeperWatcher(363): hconnection 0x1c1eceb-0x139db080e690006 Received Disconnected from ZooKeeper, ignoring
> 2012-09-18 20:18:31,231 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(363): unittest-0x139db080e69000e Received Disconnected from ZooKeeper, ignoring
> 2012-09-18 20:18:31,237 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@1f09a31] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:34950, storageID=DS-1597293543-140.211.11.27-34950-1347999501880, infoPort=40910, ipcPort=42169):DataXceiveServer:java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:31,285 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.leaseChecker] regionserver.Leases(123): RegionServer:4;hemera.apache.org,38814,1347999502374.leaseChecker closing leases
> 2012-09-18 20:18:31,285 INFO [RegionServer:4;hemera.apache.org,38814,1347999502374.leaseChecker] regionserver.Leases(130): RegionServer:4;hemera.apache.org,38814,1347999502374.leaseChecker closed leases
> 2012-09-18 20:18:32,237 WARN [pool-1-thread-1] util.MBeans(73): Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-809266774
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-809266774
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2067)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:799)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:503)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:752)
> at org.apache.hadoop.hbase.TestDrainingServer.tearDownAfterClass(TestDrainingServer.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:32,238 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down.
> 2012-09-18 20:18:32,238 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:32,341 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2a5ab9] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:46013, storageID=DS-1826840059-140.211.11.27-46013-1347999501721, infoPort=33986, ipcPort=42754):DataXceiveServer:java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:33,341 WARN [pool-1-thread-1] util.MBeans(73): Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId1641970170
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId1641970170
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2067)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:799)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:503)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:752)
> at org.apache.hadoop.hbase.TestDrainingServer.tearDownAfterClass(TestDrainingServer.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:33,342 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down.
> 2012-09-18 20:18:33,342 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:33,444 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@18c458] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:39857, storageID=DS-635958744-140.211.11.27-39857-1347999501558, infoPort=49006, ipcPort=39278):DataXceiveServer:java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:34,444 WARN [pool-1-thread-1] util.MBeans(73): Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId389747850
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId389747850
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2067)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:799)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:503)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:752)
> at org.apache.hadoop.hbase.TestDrainingServer.tearDownAfterClass(TestDrainingServer.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:34,445 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down.
> 2012-09-18 20:18:34,445 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:34,547 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@11c55bb] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:47094, storageID=DS-1080581881-140.211.11.27-47094-1347999501341, infoPort=51461, ipcPort=36418):DataXceiveServer:java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:35,547 WARN [pool-1-thread-1] util.MBeans(73): Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-1857067045
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-1857067045
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2067)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:799)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:503)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:752)
> at org.apache.hadoop.hbase.TestDrainingServer.tearDownAfterClass(TestDrainingServer.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:35,548 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down.
> 2012-09-18 20:18:35,548 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:35,650 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@b3b6a6] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:58813, storageID=DS-1129259921-140.211.11.27-58813-1347999501165, infoPort=47167, ipcPort=40188):DataXceiveServer:java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
> at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:36,179 WARN [DataNode: [/x1/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/d1b07c00-61dd-47e9-ac9b-95a060d6795d/dfscluster_6ef910b6-2888-4dda-84f2-de311d45c6a2/dfs/data/data1,/x1/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/d1b07c00-61dd-47e9-ac9b-95a060d6795d/dfscluster_6ef910b6-2888-4dda-84f2-de311d45c6a2/dfs/data/data2]] util.MBeans(73): Hadoop:service=DataNode,name=DataNodeInfo
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=DataNodeInfo
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.unRegisterMXBean(DataNode.java:522)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:737)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1471)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:36,650 WARN [pool-1-thread-1] util.MBeans(73): Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-245364923
> javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-UndefinedStorageId-245364923
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2067)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:799)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:566)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:550)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:503)
> at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:752)
> at org.apache.hadoop.hbase.TestDrainingServer.tearDownAfterClass(TestDrainingServer.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-09-18 20:18:36,651 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down.
> 2012-09-18 20:18:36,651 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
> 2012-09-18 20:18:36,753 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@1827d1] namenode.FSNamesystem$ReplicationMonitor(2718): ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted
> 2012-09-18 20:18:36,774 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(756): Minicluster is down
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira