You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (JIRA)" <ji...@apache.org> on 2014/01/16 02:42:20 UTC
[jira] [Comment Edited] (HBASE-10349) Table became unusable when
master balanced its region after table was dropped
[ https://issues.apache.org/jira/browse/HBASE-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872891#comment-13872891 ]
Ted Yu edited comment on HBASE-10349 at 1/16/14 1:42 AM:
---------------------------------------------------------
In the master log:
{code}
2014-01-15 09:41:42,395 INFO [AM.ZK.Worker-pool2-t857] master.RegionStates: Offlined afc82d1ceabbaca36a504b83b65fc0c9 from hor15n07.gq1.ygridcore.net,60020,1389770493951
{code}
This should correspond to this line in regionOffline():
{code}
LOG.info("Offlined " + hri.getShortNameToLog() + " from " + oldServerName);
Set<HRegionInfo> oldRegions = serverHoldings.get(oldServerName);
oldRegions.remove(hri);
{code}
But if hri was removed from oldRegions, I don't see why master would issue balance to the same region: afc82d1ceabbaca36a504b83b65fc0c9
was (Author: yuzhihong@gmail.com):
>From namenode log I don't see the deletion corresponding to what Matteo posted @ 2014-01-15 09:41:43,639 :
{code}
2014-01-15 09:41:42,322 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(2873)) - BLOCK* allocateBlock: /apps/hbase/data/data/default/tablethree_mod/ afc82d1ceabbaca36a504b83b65fc0c9/.tmp/2281cdbf37c54ab3b8dcc57069cae993. BP-1099602576-68.142.247.161-1389770223091 blk_1073743631_2807{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[68.142.246.154:50010|RBW], ReplicaUnderConstruction[68.142.246.158:50010|RBW], ReplicaUnderConstruction[68.142.246.159:50010|RBW]]}
...
2014-01-15 09:41:42,344 INFO hdfs.StateChange (FSNamesystem.java:completeFile(2814)) - DIR* completeFile: /apps/hbase/data/data/default/tablethree_mod/ afc82d1ceabbaca36a504b83b65fc0c9/.tmp/2281cdbf37c54ab3b8dcc57069cae993 is closed by DFSClient_hb_rs_hor15n07.gq1.ygridcore.net,60020,1389770493951_719601563_28
...
2014-01-15 09:41:44,868 INFO hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(2873)) - BLOCK* allocateBlock: /apps/hbase/data/data/default/tablethree_mod/ afc82d1ceabbaca36a504b83b65fc0c9/.tmp/.regioninfo. BP-1099602576-68.142.247.161-1389770223091 blk_1073743636_2812{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[68.142.246.159:50010|RBW], ReplicaUnderConstruction[68.142.246.156:50010|RBW], ReplicaUnderConstruction[68.142.246.154:50010|RBW]]}
{code}
> Table became unusable when master balanced its region after table was dropped
> -----------------------------------------------------------------------------
>
> Key: HBASE-10349
> URL: https://issues.apache.org/jira/browse/HBASE-10349
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Attachments: 10349-hadoop-hdfs-namenode-hor11n14.gq1.ygridcore.net.zip, 10349-output.log, hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
>
>
> 0.98 was used.
> This was sequence of events:
> create 'tablethree_mod'
> snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
> disable 'tablethree_mod'
> 2014-01-15 09:34:51,749 restore_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:35:07,210 enable 'tablethree_mod'
> 2014-01-15 09:35:46,134 delete_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:41:42,210 disable 'tablethree_mod'
> 2014-01-15 09:41:43,610 drop 'tablethree_mod'
> create 'tablethree_mod'
> For the last table creation request:
> {code}
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
> {NAME => 'f1', VERSIONS => 3}
> ,
> {NAME => 'f2', VERSIONS => 3}
> ,
> {NAME => 'f3', VERSIONS => 3}
> '
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '0', 'f1:q1', 'value-0', 10'
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '1', 'f1:q1', 'value-1', 20'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '2', 'f2:q2', 'value-2', 30'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '3', 'f3:q3', 'value-3', 40'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '4', 'f3:q3', 'value-4', 50'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to file. Will execute them now.
> 2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase shell /grid/0/tmp/hwqe/artifacts/tmp-471142
> 2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN [main] conf.Configuration: hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> 2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists: tablethree_mod!
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> {code}
> This was an intermittent issue after using Snapshots, a table is not properly dropped / and not able to properly re-create with the same name. And a HRegion is empty or null Error occurs. (When you try to drop the table it says it does not exist, and when you try to create the table it says that it does already exist).
> {code}
> 2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or empty in hbase:meta, row=keyvalues=
> {tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0, tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0, tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0}
> {code}
> Thanks to Huned who discovered this issue.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)