You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Dave Latham (JIRA)" <ji...@apache.org> on 2010/08/05 18:39:17 UTC
[jira] Commented: (HBASE-2812) Disable 'table' fails to complete
frustrating my ability to test easily
[ https://issues.apache.org/jira/browse/HBASE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895726#action_12895726 ]
Dave Latham commented on HBASE-2812:
------------------------------------
I have also run into this, using 0.20.6. I have a test environment that runs a set of automated tests with our latest code every hour. Some of these tests are set to disable, drop, and recreate a table in order to make an empty one. After succeeding for 5 to 10 executions, it seems to get into this state. I'm not able to disable and get rid of the test table until hbase is restarted, so it is definitely hampering our ability to do automated testing with hbase.
The logs I have look similar, though the master and region server are separate processes with separate logs in my environment.
Does anyone have any ideas as to the cause of this? Is there any more information that would help in diagnosing?
> Disable 'table' fails to complete frustrating my ability to test easily
> -----------------------------------------------------------------------
>
> Key: HBASE-2812
> URL: https://issues.apache.org/jira/browse/HBASE-2812
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.20.6, 0.89.20100621
> Environment: 0.89, non-distributed mode
> Reporter: Sam Pullara
> Fix For: 0.20.7, 0.90.0
>
>
> I see this in the client after it gives up:
> hbase(main):006:0> disable 'test_schema'
> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took too long to wait for the table test_schema to be disabled.
> Here is some help for this command:
> Disable the named table: e.g. "hbase> disable 't1'"
> and this in the server log, a set of about 5 reports it is closing per disable call:
> 2010-07-03 15:19:47,554 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:47,554 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:47,555 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:47,576 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:47,576 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,567 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:48,567 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:48,568 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:48,577 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,578 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:49,580 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:49,580 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:49,581 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:50,580 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,592 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:50,592 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:50,593 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:51,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:51,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:52,605 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:52,605 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:52,606 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:52,703 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 3.0
> 2010-07-03 15:19:52,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:19:52,867 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:19:53,585 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,585 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=52, Hit=50, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.15384340286255%, Miss Ratio=3.8461539894342422%, Evicted/Run=NaN
> 2010-07-03 15:19:54,617 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:54,617 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:54,618 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:54,821 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
> 2010-07-03 15:19:55,589 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:55,589 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:58,629 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing unserved regions
> 2010-07-03 15:19:58,629 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing regions currently being served
> 2010-07-03 15:19:58,630 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:59,595 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:59,595 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:20:52,706 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 3.0
> 2010-07-03 15:20:52,866 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:20:52,873 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:20:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=54, Hit=52, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.29629850387573%, Miss Ratio=3.7037037312984467%, Evicted/Run=NaN
> 2010-07-03 15:20:54,823 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
> 2010-07-03 15:21:52,709 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 3.0
> 2010-07-03 15:21:52,869 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>}
> 2010-07-03 15:21:52,876 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey: <>} complete
> 2010-07-03 15:21:53,672 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952), Counts: Blocks=2, Access=56, Hit=54, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.42857313156128%, Miss Ratio=3.57142873108387%, Evicted/Run=NaN
> 2010-07-03 15:21:54,826 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>}
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey: <>} complete
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.