You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2014/07/17 01:17:05 UTC
[jira] [Resolved] (HBASE-3190) Problem with disabling and droping
table
[ https://issues.apache.org/jira/browse/HBASE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-3190.
-----------------------------------
Resolution: Cannot Reproduce
Reopen or file new issue is still relevant with modern HBase versions
> Problem with disabling and droping table
> ----------------------------------------
>
> Key: HBASE-3190
> URL: https://issues.apache.org/jira/browse/HBASE-3190
> Project: HBase
> Issue Type: Bug
> Reporter: Sebastian Bauer
>
> Table disabling was interrupted by kill -9 all part of hbase and now we cannot do anything with this table, disabling doesn't show any exception:
> hbase(main):019:0> disable 'NGolden_CTU'
> 0 row(s) in 0.0250 seconds
> but droping show this:
> hbase(main):020:0> drop 'NGolden_CTU'
> ERROR: org.apache.hadoop.hbase.TableNotDisabledException: org.apache.hadoop.hbase.TableNotDisabledException: NGolden_CTU
> at org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:861)
> at org.apache.hadoop.hbase.master.handler.TableEventHandler.<init>(TableEventHandler.java:52)
> at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:42)
> at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:779)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> Here is some help for this command:
> Drop the named table. Table must first be disabled. If table has
> more than one region, run a major compaction on .META.:
> hbase> major_compact ".META."
> after this nothing strange is in logs
> when we restart hbase we get this:
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780005 Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:37,892 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_OPEN_REGION
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
> at org.apache.hadoop.hbase.executor.RegionTransitionData.fromBytes(RegionTransitionData.java:198)
> at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:669)
> at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:549)
> at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:542)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:208)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:89)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Other logs with this region:
> hbase@db2a:logs$ grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-master-db2a.goldenline.pl.log
> 2010-11-03 08:54:02,575 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002 Async create of unassigned node for 0c8579e52b0ea3f2dab5b6a857ad030b with OFFLINE state
> 2010-11-03 08:54:03,555 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback: rs=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. state=OFFLINE, ts=1288770842575, server=db2a.goldenline.pl,60020,1288770551154
> 2010-11-03 08:54:03,777 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback: rs=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. state=OFFLINE, ts=1288770842575
> 2010-11-03 08:54:42,836 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. state=PENDING_OPEN, ts=1288770843777
> 2010-11-03 08:54:42,836 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_OPEN for too long, reassigning region=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. state=PENDING_OPEN, ts=1288770843777
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. so generated a random one; hri=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b., src=, dest=db2b.goldenline.pl,60020,1288770553679; 2 (online=2, exclude=null) available servers
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. to db2b.goldenline.pl,60020,1288770553679
> 2010-11-03 08:56:12,824 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=db2a.goldenline.pl,60020,1288770551154, region=0c8579e52b0ea3f2dab5b6a857ad030b
> 2010-11-03 08:56:12,975 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=db2a.goldenline.pl,60020,1288770551154, region=0c8579e52b0ea3f2dab5b6a857ad030b
> 2010-11-03 08:56:12,976 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for 0c8579e52b0ea3f2dab5b6a857ad030b; deleting unassigned node
> 2010-11-03 08:56:12,976 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002 Deleting existing unassigned node for 0c8579e52b0ea3f2dab5b6a857ad030b that is in expected state RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,978 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002 Successfully deleted unassigned node for region 0c8579e52b0ea3f2dab5b6a857ad030b in expected state RS_ZK_REGION_OPENED
> 2010-11-03 08:56:13,011 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. on db2a.goldenline.pl,60020,1288770551154
> hbase@db2a:logs$ grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-regionserver-db2a.goldenline.pl.log
> 2010-11-03 08:54:04,470 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,770 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,770 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.', STARTKEY => '3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E', ENDKEY => '3065-d_2010_9_11_47D955785DEC3CE95377D70F67BA5ABC', ENCODED => 0c8579e52b0ea3f2dab5b6a857ad030b, TABLE => {{NAME => 'NGolden_CTU', FAMILIES => [{NAME => 'c', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,939 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded row bloom filter metadata for hdfs://db2a:50001/hbase/NGolden_CTU/0c8579e52b0ea3f2dab5b6a857ad030b/c/561636411807463667
> 2010-11-03 08:56:12,939 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://db2a:50001/hbase/NGolden_CTU/0c8579e52b0ea3f2dab5b6a857ad030b/c/561636411807463667, isReference=false, isBulkLoadResult=false, seqid=1820373362, majorCompaction=true
> 2010-11-03 08:56:12,939 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.; next sequenceid=1820373363
> 2010-11-03 08:56:12,939 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,970 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,971 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b. in region .META.,,1 with server=db2a.goldenline.pl:60020, startcode=1288770551154
> 2010-11-03 08:56:12,971 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,973 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001 Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,973 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> hbase@db2b:logs$ grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-regionserver-db2b.goldenline.pl.log
> 2010-11-03 08:54:42,840 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780005 Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
--
This message was sent by Atlassian JIRA
(v6.2#6252)