You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 唐 颖 <iv...@gmail.com> on 2012/10/16 09:52:52 UTC
hbase can't drop a table
I disable this table ivytest_deu , drop it .Error occurs.
ERROR: java.io.IOException: java.io.IOException: HTableDescriptor missing for ivytest_deu
at org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescriptor(TableEventHandler.java:174)
at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:44)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
Here is some help for this command:
Drop the named table. Table must first be disabled. If table has
more than one region, run a major compaction on .META.:
hbase> major_compact ".META."
The major_compact ".META." doesn't work.
Then i try to create it ,but HBase says it .
ERROR: Table already exists: ivytest_deu!
After checking the region server log , the region server is always trying to load this region.
2012-10-16 00:00:00,308 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
2012-10-16 00:00:00,313 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting it: deu_ivytest
2012-10-16 00:00:00,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,}
2012-10-16 00:00:00,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol handler: region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45. protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggregationProtocol
2012-10-16 00:00:00,358 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45., starting to roll back the global memstore size.
2012-10-16 00:00:00,358 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
And we have a endpoint in base .After the base tried to load this table ivy test_deu for 90,000 times ,the endpoint class also has been loaded for 90,000 times.
The jvm memory has been filled.
The gcutil shows
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
The jmap dump file shows
3982039 instances of class org.apache.hadoop.hbase.KeyValue
191050 instances of class org.apache.hadoop.fs.Path
187364 instances of class org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
187301 instances of class org.cliffc.high_scale_lib.Counter
102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
93652 instances of class org.apache.hadoop.hbase.HRegionInfo
93650 instances of class com.google.common.collect.MutableClassToInstanceMap
93650 instances of class DEUColumnAggregationEndpoint
DEUColumnAggregationEndpoint is our endpoint class.
We guess the 90,000 times check this table and load endpoint class leads this memory leak.
But how to drop this table?
RE: hbase can't drop a table
Posted by "Ramkrishna.S.Vasudevan" <ra...@huawei.com>.
Which version of HBase?
The below logs that you have attached says about a different table right '
deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
And the one you are trying to drop is ' ivytest_deu’
Regards
Ram
> -----Original Message-----
> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> Sent: Tuesday, October 16, 2012 1:23 PM
> To: user@hbase.apache.org
> Subject: hbase can't drop a table
>
> I disable this table ivytest_deu , drop it .Error occurs.
>
>
> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> missing for ivytest_deu
> at
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
> ptor(TableEventHandler.java:174)
> at
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
> TableHandler.java:44)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> va:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
> gine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
> 76)
>
> Here is some help for this command:
> Drop the named table. Table must first be disabled. If table has
> more than one region, run a major compaction on .META.:
>
> hbase> major_compact ".META."
>
> The major_compact ".META." doesn't work.
> Then i try to create it ,but HBase says it .
>
> ERROR: Table already exists: ivytest_deu!
>
> After checking the region server log , the region server is always
> trying to load this region.
>
> 2012-10-16 00:00:00,308 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> open region:
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> 2012-10-16 00:00:00,313 WARN
> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
> is in HBase's root directory and doesn't contain a table descriptor, do
> consider deleting it: deu_ivytest
> 2012-10-16 00:00:00,358 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY => '', ENDKEY => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,}
> 2012-10-16 00:00:00,358 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> handler:
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
> ationProtocol
> 2012-10-16 00:00:00,358 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open of
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> starting to roll back the global memstore size.
> 2012-10-16 00:00:00,358 INFO
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
> of region {NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY => '', ENDKEY => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
>
> And we have a endpoint in base .After the base tried to load this table
> ivy test_deu for 90,000 times ,the endpoint class also has been loaded
> for 90,000 times.
> The jvm memory has been filled.
> The gcutil shows
> S0C S1C S0U S1U EC EU OC OU PC
> PU YGC YGCT FGC FGCT GCT
> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>
> The jmap dump file shows
>
> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> 191050 instances of class org.apache.hadoop.fs.Path
> 187364 instances of class
> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 187301 instances of class org.cliffc.high_scale_lib.Counter
> 102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> 93650 instances of class
> com.google.common.collect.MutableClassToInstanceMap
> 93650 instances of class DEUColumnAggregationEndpoint
>
> DEUColumnAggregationEndpoint is our endpoint class.
>
> We guess the 90,000 times check this table and load endpoint class
> leads this memory leak.
>
> But how to drop this table?
>
>
>
>
>
>
>
>
RE: hbase can't drop a table
Posted by "Ramkrishna.S.Vasudevan" <ra...@huawei.com>.
What does the 'list' command show? Does it say the table exists or not?
What I can infer here is that the HTableDescriptor file got deleted but the
META is having the entry. Any chance of the HTD getting accidently deleted
in your cluster?
The hbck tool with -fixOrphanTables should atleast try to create the
HTableDescriptor file I suppose. Then restart the cluster and then see what
happens.
I will not be able to access the logs even if you add it to pastebin. But
pls do it so that some one else who has access can look into it.
Regards
Ram
> -----Original Message-----
> From: 张磊 [mailto:zhanglei@youku.com]
> Sent: Tuesday, October 16, 2012 1:44 PM
> To: 'user@hbase.apache.org'
> Subject: RE: hbase can't drop a table
>
> Hope this can help you!
> https://issues.apache.org/jira/browse/HBASE-
> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=13418790#comment-13418790
>
> Fowler Zhang
>
> -----Original Message-----
> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> Sent: 2012年10月16日 16:08
> To: user@hbase.apache.org
> Subject: Re: hbase can't drop a table
>
> version 0.94.0, r8547
>
> And the table is ivytest_deu.
>
>
> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
> <ra...@huawei.com> 写道：
>
> > Which version of HBase?
> >
> >
> > The below logs that you have attached says about a different table
> right '
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> > And the one you are trying to drop is ' ivytest_deu’
> >
> > Regards
> > Ram
> >
> >
> >
> >> -----Original Message-----
> >> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> >> Sent: Tuesday, October 16, 2012 1:23 PM
> >> To: user@hbase.apache.org
> >> Subject: hbase can't drop a table
> >>
> >> I disable this table ivytest_deu , drop it .Error occurs.
> >>
> >>
> >> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> >> missing for ivytest_deu
> >> at
> >>
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
> >> ri
> >> ptor(TableEventHandler.java:174)
> >> at
> >>
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
> >> te
> >> TableHandler.java:44)
> >> at
> >>
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> ja
> >> va:39)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> >> so
> >> rImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
> >> En
> >> gine.java:364)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> 13
> >> 76)
> >>
> >> Here is some help for this command:
> >> Drop the named table. Table must first be disabled. If table has
> more
> >> than one region, run a major compaction on .META.:
> >>
> >> hbase> major_compact ".META."
> >>
> >> The major_compact ".META." doesn't work.
> >> Then i try to create it ,but HBase says it .
> >>
> >> ERROR: Table already exists: ivytest_deu!
> >>
> >> After checking the region server log , the region server is always
> >> trying to load this region.
> >>
> >> 2012-10-16 00:00:00,308 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
> >> to open region:
> >> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> >> 2012-10-16 00:00:00,313 WARN
> >> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
> folder
> >> is in HBase's root directory and doesn't contain a table descriptor,
> >> do consider deleting it: deu_ivytest
> >> 2012-10-16 00:00:00,358 DEBUG
> >> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
> >> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> >> STARTKEY => '', ENDKEY => '', ENCODED =>
> >> 985d6ca9986d7d8cfaf82daf523fcd45,}
> >> 2012-10-16 00:00:00,358 DEBUG
> >> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> >> handler:
> >> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> >>
> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
> >> eg
> >> ationProtocol
> >> 2012-10-16 00:00:00,358 ERROR
> >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> >> Failed open of
> >> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> >> starting to roll back the global memstore size.
> >> 2012-10-16 00:00:00,358 INFO
> >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> >> Opening of region {NAME =>
> >> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> >> STARTKEY => '', ENDKEY => '', ENCODED =>
> >> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in
> >> ZK
> >>
> >> And we have a endpoint in base .After the base tried to load this
> >> table ivy test_deu for 90,000 times ,the endpoint class also has
> been
> >> loaded for 90,000 times.
> >> The jvm memory has been filled.
> >> The gcutil shows
> >> S0C S1C S0U S1U EC EU OC OU
> PC
> >> PU YGC YGCT FGC FGCT GCT
> >> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> >> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> 2792768.0
> >> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> >>
> >> The jmap dump file shows
> >>
> >> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> >> 191050 instances of class org.apache.hadoop.fs.Path
> >> 187364 instances of class
> >> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> >> 187301 instances of class org.cliffc.high_scale_lib.Counter
> >> 102272 instances of class
> net.sf.ehcache.concurrent.ReadWriteLockSync
> >> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> >> 93650 instances of class
> >> com.google.common.collect.MutableClassToInstanceMap
> >> 93650 instances of class DEUColumnAggregationEndpoint
> >>
> >> DEUColumnAggregationEndpoint is our endpoint class.
> >>
> >> We guess the 90,000 times check this table and load endpoint class
> >> leads this memory leak.
> >>
> >> But how to drop this table?
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
RE: hbase can't drop a table
Posted by "Ramkrishna.S.Vasudevan" <ra...@huawei.com>.
Thanks for the write up Kevin. Many users will be benefited by this.
Regards
Ram
> -----Original Message-----
> From: Kevin O'dell [mailto:kevin.odell@cloudera.com]
> Sent: Tuesday, October 16, 2012 9:42 PM
> To: user@hbase.apache.org
> Subject: Re: hbase can't drop a table
>
> If you get in that situation again:
>
> 1.) Verify that you don't have any remnants of the tables in HDFS
> hadoop fs -ls /hbase/
>
> 2.) If you do have any remnants and you don't care about these tables
> hadoop fs -mv /hbase/<table_name> /tmp
>
> 3.) ./bin/hbase hbck -fixMeta -fixAssignments
>
> This should clean up META to match your HDFS and allow you to move
> forward.
> If not you may need a restart of HBase in case you had a RIT or
> something
> like that hanging around.
>
> On Tue, Oct 16, 2012 at 5:06 AM, 唐 颖 <iv...@gmail.com> wrote:
>
> > After checking the .META. table , the ivy test_deu and deu_ivytest
> entries
> > do exist .
> >
> > ROW COLUMN+CELL
> > ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=info:regioninfo, timestamp=1348821682041, value={NAME =>
> > 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.',
> STARTKEY =>
> > '', ENDKEY
> > 4c3e0277c. => '', ENCODED
> =>
> > 77eb091b4753dd3b713f29c4c3e0277c,}
> > ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:server,
> > timestamp=1350032968219, value=ELEX-LA-WEB10:61020
> > 4c3e0277c.
> > ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=info:serverstartcode, timestamp=1350032968219,
> value=1350032549030
> > 4c3e0277c.
> >
> > ROW COLUMN+CELL
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=info:regioninfo, timestamp=1348826121970, value={NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY =>
> > '', ENDKEY
> > f523fcd45. => '', ENCODED
> =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,}
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:server,
> > timestamp=1348826122164, value=ELEX-LA-WEB10:61020
> > f523fcd45.
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=info:serverstartcode, timestamp=1348826122164,
> value=1348648468768
> > f523fcd45.
> >
> > And i deleted them from .META. table.Things seem Ok.
> >
> > The region server won't try to load these regions.
> >
> > Yes,it seems that the HTableDescriptor file got deleted but the META
> is
> > having the entry.
> >
> >
> > Thanks!
> >
> >
> > 在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <
> > ramkrishna.vasudevan@huawei.com> 写道：
> >
> > > What does the 'list' command show? Does it say the table exists or
> not?
> > >
> > > What I can infer here is that the HTableDescriptor file got
> deleted but
> > the
> > > META is having the entry. Any chance of the HTD getting accidently
> > deleted
> > > in your cluster?
> > >
> > > The hbck tool with -fixOrphanTables should atleast try to create
> the
> > > HTableDescriptor file I suppose. Then restart the cluster and then
> see
> > what
> > > happens.
> > > I will not be able to access the logs even if you add it to
> pastebin.
> > But
> > > pls do it so that some one else who has access can look into it.
> > >
> > > Regards
> > > Ram
> > >> -----Original Message-----
> > >> From: 张磊 [mailto:zhanglei@youku.com]
> > >> Sent: Tuesday, October 16, 2012 1:44 PM
> > >> To: 'user@hbase.apache.org'
> > >> Subject: RE: hbase can't drop a table
> > >>
> > >> Hope this can help you!
> > >> https://issues.apache.org/jira/browse/HBASE-
> > >> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> > >> tabpanel&focusedCommentId=13418790#comment-13418790
> > >>
> > >> Fowler Zhang
> > >>
> > >> -----Original Message-----
> > >> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> > >> Sent: 2012年10月16日 16:08
> > >> To: user@hbase.apache.org
> > >> Subject: Re: hbase can't drop a table
> > >>
> > >> version 0.94.0, r8547
> > >>
> > >> And the table is ivytest_deu.
> > >>
> > >>
> > >> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
> > >> <ra...@huawei.com> 写道：
> > >>
> > >>> Which version of HBase?
> > >>>
> > >>>
> > >>> The below logs that you have attached says about a different
> table
> > >> right '
> > >>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> > >>> And the one you are trying to drop is ' ivytest_deu’
> > >>>
> > >>> Regards
> > >>> Ram
> > >>>
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> > >>>> Sent: Tuesday, October 16, 2012 1:23 PM
> > >>>> To: user@hbase.apache.org
> > >>>> Subject: hbase can't drop a table
> > >>>>
> > >>>> I disable this table ivytest_deu , drop it .Error occurs.
> > >>>>
> > >>>>
> > >>>> ERROR: java.io.IOException: java.io.IOException:
> HTableDescriptor
> > >>>> missing for ivytest_deu
> > >>>> at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
> > >>>> ri
> > >>>> ptor(TableEventHandler.java:174)
> > >>>> at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
> > >>>> te
> > >>>> TableHandler.java:44)
> > >>>> at
> > >>>>
> > >>
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > >>>> at
> > >>>>
> > >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> > >>>> ja
> > >>>> va:39)
> > >>>> at
> > >>>>
> > >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> > >>>> so
> > >>>> rImpl.java:25)
> > >>>> at java.lang.reflect.Method.invoke(Method.java:597)
> > >>>> at
> > >>>>
> > >>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
> > >>>> En
> > >>>> gine.java:364)
> > >>>> at
> > >>>>
> > >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> > >>>> 13
> > >>>> 76)
> > >>>>
> > >>>> Here is some help for this command:
> > >>>> Drop the named table. Table must first be disabled. If table has
> > >> more
> > >>>> than one region, run a major compaction on .META.:
> > >>>>
> > >>>> hbase> major_compact ".META."
> > >>>>
> > >>>> The major_compact ".META." doesn't work.
> > >>>> Then i try to create it ,but HBase says it .
> > >>>>
> > >>>> ERROR: Table already exists: ivytest_deu!
> > >>>>
> > >>>> After checking the region server log , the region server is
> always
> > >>>> trying to load this region.
> > >>>>
> > >>>> 2012-10-16 00:00:00,308 INFO
> > >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received
> request
> > >>>> to open region:
> > >>>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > >>>> 2012-10-16 00:00:00,313 WARN
> > >>>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
> > >> folder
> > >>>> is in HBase's root directory and doesn't contain a table
> descriptor,
> > >>>> do consider deleting it: deu_ivytest
> > >>>> 2012-10-16 00:00:00,358 DEBUG
> > >>>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
> {NAME
> > >>>> =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > >>>> STARTKEY => '', ENDKEY => '', ENCODED =>
> > >>>> 985d6ca9986d7d8cfaf82daf523fcd45,}
> > >>>> 2012-10-16 00:00:00,358 DEBUG
> > >>>> org.apache.hadoop.hbase.regionserver.HRegion: Registered
> protocol
> > >>>> handler:
> > >>>>
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > >>>>
> > >>
> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
> > >>>> eg
> > >>>> ationProtocol
> > >>>> 2012-10-16 00:00:00,358 ERROR
> > >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> > >>>> Failed open of
> > >>>>
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> > >>>> starting to roll back the global memstore size.
> > >>>> 2012-10-16 00:00:00,358 INFO
> > >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> > >>>> Opening of region {NAME =>
> > >>>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > >>>> STARTKEY => '', ENDKEY => '', ENCODED =>
> > >>>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as
> FAILED_OPEN in
> > >>>> ZK
> > >>>>
> > >>>> And we have a endpoint in base .After the base tried to load
> this
> > >>>> table ivy test_deu for 90,000 times ,the endpoint class also has
> > >> been
> > >>>> loaded for 90,000 times.
> > >>>> The jvm memory has been filled.
> > >>>> The gcutil shows
> > >>>> S0C S1C S0U S1U EC EU OC OU
> > >> PC
> > >>>> PU YGC YGCT FGC FGCT GCT
> > >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > >>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> > >>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> > >> 2792768.0
> > >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> > >>>>
> > >>>> The jmap dump file shows
> > >>>>
> > >>>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> > >>>> 191050 instances of class org.apache.hadoop.fs.Path
> > >>>> 187364 instances of class
> > >>>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> > >>>> 187301 instances of class org.cliffc.high_scale_lib.Counter
> > >>>> 102272 instances of class
> > >> net.sf.ehcache.concurrent.ReadWriteLockSync
> > >>>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> > >>>> 93650 instances of class
> > >>>> com.google.common.collect.MutableClassToInstanceMap
> > >>>> 93650 instances of class DEUColumnAggregationEndpoint
> > >>>>
> > >>>> DEUColumnAggregationEndpoint is our endpoint class.
> > >>>>
> > >>>> We guess the 90,000 times check this table and load endpoint
> class
> > >>>> leads this memory leak.
> > >>>>
> > >>>> But how to drop this table?
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >
> > >
> >
> >
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
Re: hbase can't drop a table
Posted by Kevin O'dell <ke...@cloudera.com>.
If you get in that situation again:
1.) Verify that you don't have any remnants of the tables in HDFS
hadoop fs -ls /hbase/
2.) If you do have any remnants and you don't care about these tables
hadoop fs -mv /hbase/<table_name> /tmp
3.) ./bin/hbase hbck -fixMeta -fixAssignments
This should clean up META to match your HDFS and allow you to move forward.
If not you may need a restart of HBase in case you had a RIT or something
like that hanging around.
On Tue, Oct 16, 2012 at 5:06 AM, 唐 颖 <iv...@gmail.com> wrote:
> After checking the .META. table , the ivy test_deu and deu_ivytest entries
> do exist .
>
> ROW COLUMN+CELL
> ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:regioninfo, timestamp=1348821682041, value={NAME =>
> 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.', STARTKEY =>
> '', ENDKEY
> 4c3e0277c. => '', ENCODED =>
> 77eb091b4753dd3b713f29c4c3e0277c,}
> ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:server,
> timestamp=1350032968219, value=ELEX-LA-WEB10:61020
> 4c3e0277c.
> ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:serverstartcode, timestamp=1350032968219, value=1350032549030
> 4c3e0277c.
>
> ROW COLUMN+CELL
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:regioninfo, timestamp=1348826121970, value={NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY =>
> '', ENDKEY
> f523fcd45. => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,}
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:server,
> timestamp=1348826122164, value=ELEX-LA-WEB10:61020
> f523fcd45.
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:serverstartcode, timestamp=1348826122164, value=1348648468768
> f523fcd45.
>
> And i deleted them from .META. table.Things seem Ok.
>
> The region server won't try to load these regions.
>
> Yes,it seems that the HTableDescriptor file got deleted but the META is
> having the entry.
>
>
> Thanks!
>
>
> 在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <
> ramkrishna.vasudevan@huawei.com> 写道：
>
> > What does the 'list' command show? Does it say the table exists or not?
> >
> > What I can infer here is that the HTableDescriptor file got deleted but
> the
> > META is having the entry. Any chance of the HTD getting accidently
> deleted
> > in your cluster?
> >
> > The hbck tool with -fixOrphanTables should atleast try to create the
> > HTableDescriptor file I suppose. Then restart the cluster and then see
> what
> > happens.
> > I will not be able to access the logs even if you add it to pastebin.
> But
> > pls do it so that some one else who has access can look into it.
> >
> > Regards
> > Ram
> >> -----Original Message-----
> >> From: 张磊 [mailto:zhanglei@youku.com]
> >> Sent: Tuesday, October 16, 2012 1:44 PM
> >> To: 'user@hbase.apache.org'
> >> Subject: RE: hbase can't drop a table
> >>
> >> Hope this can help you!
> >> https://issues.apache.org/jira/browse/HBASE-
> >> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> >> tabpanel&focusedCommentId=13418790#comment-13418790
> >>
> >> Fowler Zhang
> >>
> >> -----Original Message-----
> >> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> >> Sent: 2012年10月16日 16:08
> >> To: user@hbase.apache.org
> >> Subject: Re: hbase can't drop a table
> >>
> >> version 0.94.0, r8547
> >>
> >> And the table is ivytest_deu.
> >>
> >>
> >> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
> >> <ra...@huawei.com> 写道：
> >>
> >>> Which version of HBase?
> >>>
> >>>
> >>> The below logs that you have attached says about a different table
> >> right '
> >>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> >>> And the one you are trying to drop is ' ivytest_deu’
> >>>
> >>> Regards
> >>> Ram
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
> >>>> Sent: Tuesday, October 16, 2012 1:23 PM
> >>>> To: user@hbase.apache.org
> >>>> Subject: hbase can't drop a table
> >>>>
> >>>> I disable this table ivytest_deu , drop it .Error occurs.
> >>>>
> >>>>
> >>>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> >>>> missing for ivytest_deu
> >>>> at
> >>>>
> >> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
> >>>> ri
> >>>> ptor(TableEventHandler.java:174)
> >>>> at
> >>>>
> >> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
> >>>> te
> >>>> TableHandler.java:44)
> >>>> at
> >>>>
> >> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>> ja
> >>>> va:39)
> >>>> at
> >>>>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> >>>> so
> >>>> rImpl.java:25)
> >>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> at
> >>>>
> >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
> >>>> En
> >>>> gine.java:364)
> >>>> at
> >>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>> 13
> >>>> 76)
> >>>>
> >>>> Here is some help for this command:
> >>>> Drop the named table. Table must first be disabled. If table has
> >> more
> >>>> than one region, run a major compaction on .META.:
> >>>>
> >>>> hbase> major_compact ".META."
> >>>>
> >>>> The major_compact ".META." doesn't work.
> >>>> Then i try to create it ,but HBase says it .
> >>>>
> >>>> ERROR: Table already exists: ivytest_deu!
> >>>>
> >>>> After checking the region server log , the region server is always
> >>>> trying to load this region.
> >>>>
> >>>> 2012-10-16 00:00:00,308 INFO
> >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
> >>>> to open region:
> >>>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> >>>> 2012-10-16 00:00:00,313 WARN
> >>>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
> >> folder
> >>>> is in HBase's root directory and doesn't contain a table descriptor,
> >>>> do consider deleting it: deu_ivytest
> >>>> 2012-10-16 00:00:00,358 DEBUG
> >>>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
> >>>> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> >>>> STARTKEY => '', ENDKEY => '', ENCODED =>
> >>>> 985d6ca9986d7d8cfaf82daf523fcd45,}
> >>>> 2012-10-16 00:00:00,358 DEBUG
> >>>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> >>>> handler:
> >>>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> >>>>
> >> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
> >>>> eg
> >>>> ationProtocol
> >>>> 2012-10-16 00:00:00,358 ERROR
> >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> >>>> Failed open of
> >>>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> >>>> starting to roll back the global memstore size.
> >>>> 2012-10-16 00:00:00,358 INFO
> >>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
> >>>> Opening of region {NAME =>
> >>>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> >>>> STARTKEY => '', ENDKEY => '', ENCODED =>
> >>>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in
> >>>> ZK
> >>>>
> >>>> And we have a endpoint in base .After the base tried to load this
> >>>> table ivy test_deu for 90,000 times ,the endpoint class also has
> >> been
> >>>> loaded for 90,000 times.
> >>>> The jvm memory has been filled.
> >>>> The gcutil shows
> >>>> S0C S1C S0U S1U EC EU OC OU
> >> PC
> >>>> PU YGC YGCT FGC FGCT GCT
> >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> >>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> >>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
> >> 2792768.0
> >>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> >>>>
> >>>> The jmap dump file shows
> >>>>
> >>>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> >>>> 191050 instances of class org.apache.hadoop.fs.Path
> >>>> 187364 instances of class
> >>>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> >>>> 187301 instances of class org.cliffc.high_scale_lib.Counter
> >>>> 102272 instances of class
> >> net.sf.ehcache.concurrent.ReadWriteLockSync
> >>>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> >>>> 93650 instances of class
> >>>> com.google.common.collect.MutableClassToInstanceMap
> >>>> 93650 instances of class DEUColumnAggregationEndpoint
> >>>>
> >>>> DEUColumnAggregationEndpoint is our endpoint class.
> >>>>
> >>>> We guess the 90,000 times check this table and load endpoint class
> >>>> leads this memory leak.
> >>>>
> >>>> But how to drop this table?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >
> >
>
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: hbase can't drop a table
Posted by 唐 颖 <iv...@gmail.com>.
After checking the .META. table , the ivy test_deu and deu_ivytest entries do exist .
ROW COLUMN+CELL
ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:regioninfo, timestamp=1348821682041, value={NAME => 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.', STARTKEY => '', ENDKEY
4c3e0277c. => '', ENCODED => 77eb091b4753dd3b713f29c4c3e0277c,}
ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:server, timestamp=1350032968219, value=ELEX-LA-WEB10:61020
4c3e0277c.
ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:serverstartcode, timestamp=1350032968219, value=1350032549030
4c3e0277c.
ROW COLUMN+CELL
deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:regioninfo, timestamp=1348826121970, value={NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY
f523fcd45. => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,}
deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:server, timestamp=1348826122164, value=ELEX-LA-WEB10:61020
f523fcd45.
deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:serverstartcode, timestamp=1348826122164, value=1348648468768
f523fcd45.
And i deleted them from .META. table.Things seem Ok.
The region server won't try to load these regions.
Yes,it seems that the HTableDescriptor file got deleted but the META is having the entry.
Thanks!
在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <ra...@huawei.com> 写道：
> What does the 'list' command show? Does it say the table exists or not?
>
> What I can infer here is that the HTableDescriptor file got deleted but the
> META is having the entry. Any chance of the HTD getting accidently deleted
> in your cluster?
>
> The hbck tool with -fixOrphanTables should atleast try to create the
> HTableDescriptor file I suppose. Then restart the cluster and then see what
> happens.
> I will not be able to access the logs even if you add it to pastebin. But
> pls do it so that some one else who has access can look into it.
>
> Regards
> Ram
>> -----Original Message-----
>> From: 张磊 [mailto:zhanglei@youku.com]
>> Sent: Tuesday, October 16, 2012 1:44 PM
>> To: 'user@hbase.apache.org'
>> Subject: RE: hbase can't drop a table
>>
>> Hope this can help you!
>> https://issues.apache.org/jira/browse/HBASE-
>> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel&focusedCommentId=13418790#comment-13418790
>>
>> Fowler Zhang
>>
>> -----Original Message-----
>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
>> Sent: 2012年10月16日 16:08
>> To: user@hbase.apache.org
>> Subject: Re: hbase can't drop a table
>>
>> version 0.94.0, r8547
>>
>> And the table is ivytest_deu.
>>
>>
>> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
>> <ra...@huawei.com> 写道：
>>
>>> Which version of HBase?
>>>
>>>
>>> The below logs that you have attached says about a different table
>> right '
>>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
>>> And the one you are trying to drop is ' ivytest_deu’
>>>
>>> Regards
>>> Ram
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
>>>> Sent: Tuesday, October 16, 2012 1:23 PM
>>>> To: user@hbase.apache.org
>>>> Subject: hbase can't drop a table
>>>>
>>>> I disable this table ivytest_deu , drop it .Error occurs.
>>>>
>>>>
>>>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
>>>> missing for ivytest_deu
>>>> at
>>>>
>> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
>>>> ri
>>>> ptor(TableEventHandler.java:174)
>>>> at
>>>>
>> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
>>>> te
>>>> TableHandler.java:44)
>>>> at
>>>>
>> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>> ja
>>>> va:39)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>>> so
>>>> rImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>>>> En
>>>> gine.java:364)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>>> 13
>>>> 76)
>>>>
>>>> Here is some help for this command:
>>>> Drop the named table. Table must first be disabled. If table has
>> more
>>>> than one region, run a major compaction on .META.:
>>>>
>>>> hbase> major_compact ".META."
>>>>
>>>> The major_compact ".META." doesn't work.
>>>> Then i try to create it ,but HBase says it .
>>>>
>>>> ERROR: Table already exists: ivytest_deu!
>>>>
>>>> After checking the region server log , the region server is always
>>>> trying to load this region.
>>>>
>>>> 2012-10-16 00:00:00,308 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
>>>> to open region:
>>>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>>>> 2012-10-16 00:00:00,313 WARN
>>>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
>> folder
>>>> is in HBase's root directory and doesn't contain a table descriptor,
>>>> do consider deleting it: deu_ivytest
>>>> 2012-10-16 00:00:00,358 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
>>>> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>>>> STARTKEY => '', ENDKEY => '', ENCODED =>
>>>> 985d6ca9986d7d8cfaf82daf523fcd45,}
>>>> 2012-10-16 00:00:00,358 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
>>>> handler:
>>>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>>>>
>> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
>>>> eg
>>>> ationProtocol
>>>> 2012-10-16 00:00:00,358 ERROR
>>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>>>> Failed open of
>>>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
>>>> starting to roll back the global memstore size.
>>>> 2012-10-16 00:00:00,358 INFO
>>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>>>> Opening of region {NAME =>
>>>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>>>> STARTKEY => '', ENDKEY => '', ENCODED =>
>>>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in
>>>> ZK
>>>>
>>>> And we have a endpoint in base .After the base tried to load this
>>>> table ivy test_deu for 90,000 times ,the endpoint class also has
>> been
>>>> loaded for 90,000 times.
>>>> The jvm memory has been filled.
>>>> The gcutil shows
>>>> S0C S1C S0U S1U EC EU OC OU
>> PC
>>>> PU YGC YGCT FGC FGCT GCT
>>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>>>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>>>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0
>> 2792768.0
>>>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>>>>
>>>> The jmap dump file shows
>>>>
>>>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
>>>> 191050 instances of class org.apache.hadoop.fs.Path
>>>> 187364 instances of class
>>>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
>>>> 187301 instances of class org.cliffc.high_scale_lib.Counter
>>>> 102272 instances of class
>> net.sf.ehcache.concurrent.ReadWriteLockSync
>>>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
>>>> 93650 instances of class
>>>> com.google.common.collect.MutableClassToInstanceMap
>>>> 93650 instances of class DEUColumnAggregationEndpoint
>>>>
>>>> DEUColumnAggregationEndpoint is our endpoint class.
>>>>
>>>> We guess the 90,000 times check this table and load endpoint class
>>>> leads this memory leak.
>>>>
>>>> But how to drop this table?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>
>
RE: hbase can't drop a table
Posted by 张磊 <zh...@youku.com>.
Hope this can help you!
https://issues.apache.org/jira/browse/HBASE-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418790#comment-13418790
Fowler Zhang
-----Original Message-----
From: 唐 颖 [mailto:ivytang0812@gmail.com]
Sent: 2012年10月16日 16:08
To: user@hbase.apache.org
Subject: Re: hbase can't drop a table
version 0.94.0, r8547
And the table is ivytest_deu.
在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan" <ra...@huawei.com> 写道：
> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
>> -----Original Message-----
>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
>> Sent: Tuesday, October 16, 2012 1:23 PM
>> To: user@hbase.apache.org
>> Subject: hbase can't drop a table
>>
>> I disable this table ivytest_deu , drop it .Error occurs.
>>
>>
>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
>> missing for ivytest_deu
>> at
>> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
>> ri
>> ptor(TableEventHandler.java:174)
>> at
>> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
>> te
>> TableHandler.java:44)
>> at
>> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>> ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> so
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>> En
>> gine.java:364)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>> 13
>> 76)
>>
>> Here is some help for this command:
>> Drop the named table. Table must first be disabled. If table has more
>> than one region, run a major compaction on .META.:
>>
>> hbase> major_compact ".META."
>>
>> The major_compact ".META." doesn't work.
>> Then i try to create it ,but HBase says it .
>>
>> ERROR: Table already exists: ivytest_deu!
>>
>> After checking the region server log , the region server is always
>> trying to load this region.
>>
>> 2012-10-16 00:00:00,308 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
>> to open region:
>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> 2012-10-16 00:00:00,313 WARN
>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
>> is in HBase's root directory and doesn't contain a table descriptor,
>> do consider deleting it: deu_ivytest
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
>> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,}
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
>> handler:
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
>> eg
>> ationProtocol
>> 2012-10-16 00:00:00,358 ERROR
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>> Failed open of
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
>> starting to roll back the global memstore size.
>> 2012-10-16 00:00:00,358 INFO
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>> Opening of region {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in
>> ZK
>>
>> And we have a endpoint in base .After the base tried to load this
>> table ivy test_deu for 90,000 times ,the endpoint class also has been
>> loaded for 90,000 times.
>> The jvm memory has been filled.
>> The gcutil shows
>> S0C S1C S0U S1U EC EU OC OU PC
>> PU YGC YGCT FGC FGCT GCT
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>>
>> The jmap dump file shows
>>
>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
>> 191050 instances of class org.apache.hadoop.fs.Path
>> 187364 instances of class
>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
>> 187301 instances of class org.cliffc.high_scale_lib.Counter
>> 102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
>> 93650 instances of class
>> com.google.common.collect.MutableClassToInstanceMap
>> 93650 instances of class DEUColumnAggregationEndpoint
>>
>> DEUColumnAggregationEndpoint is our endpoint class.
>>
>> We guess the 90,000 times check this table and load endpoint class
>> leads this memory leak.
>>
>> But how to drop this table?
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: hbase can't drop a table
Posted by 唐 颖 <iv...@gmail.com>.
version 0.94.0, r8547
And the table is ivytest_deu.
在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan" <ra...@huawei.com> 写道：
> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
>> -----Original Message-----
>> From: 唐 颖 [mailto:ivytang0812@gmail.com]
>> Sent: Tuesday, October 16, 2012 1:23 PM
>> To: user@hbase.apache.org
>> Subject: hbase can't drop a table
>>
>> I disable this table ivytest_deu , drop it .Error occurs.
>>
>>
>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
>> missing for ivytest_deu
>> at
>> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
>> ptor(TableEventHandler.java:174)
>> at
>> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
>> TableHandler.java:44)
>> at
>> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
>> gine.java:364)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
>> 76)
>>
>> Here is some help for this command:
>> Drop the named table. Table must first be disabled. If table has
>> more than one region, run a major compaction on .META.:
>>
>> hbase> major_compact ".META."
>>
>> The major_compact ".META." doesn't work.
>> Then i try to create it ,but HBase says it .
>>
>> ERROR: Table already exists: ivytest_deu!
>>
>> After checking the region server log , the region server is always
>> trying to load this region.
>>
>> 2012-10-16 00:00:00,308 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
>> open region:
>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> 2012-10-16 00:00:00,313 WARN
>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
>> is in HBase's root directory and doesn't contain a table descriptor, do
>> consider deleting it: deu_ivytest
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,}
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
>> handler:
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
>> ationProtocol
>> 2012-10-16 00:00:00,358 ERROR
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
>> open of
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
>> starting to roll back the global memstore size.
>> 2012-10-16 00:00:00,358 INFO
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
>> of region {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
>>
>> And we have a endpoint in base .After the base tried to load this table
>> ivy test_deu for 90,000 times ,the endpoint class also has been loaded
>> for 90,000 times.
>> The jvm memory has been filled.
>> The gcutil shows
>> S0C S1C S0U S1U EC EU OC OU PC
>> PU YGC YGCT FGC FGCT GCT
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>> 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
>> 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
>>
>> The jmap dump file shows
>>
>> 3982039 instances of class org.apache.hadoop.hbase.KeyValue
>> 191050 instances of class org.apache.hadoop.fs.Path
>> 187364 instances of class
>> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
>> 187301 instances of class org.cliffc.high_scale_lib.Counter
>> 102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
>> 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
>> 93650 instances of class
>> com.google.common.collect.MutableClassToInstanceMap
>> 93650 instances of class DEUColumnAggregationEndpoint
>>
>> DEUColumnAggregationEndpoint is our endpoint class.
>>
>> We guess the 90,000 times check this table and load endpoint class
>> leads this memory leak.
>>
>> But how to drop this table?
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: hbase can't drop a table
Posted by IvyTang <iv...@gmail.com>.
both ivytest_deu and deu_ivytest have the same problem.
On Tue, Oct 16, 2012 at 3:58 PM, Ramkrishna.S.Vasudevan <
ramkrishna.vasudevan@huawei.com> wrote:
> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
> > -----Original Message-----
> > From: 唐 颖 [mailto:ivytang0812@gmail.com]
> > Sent: Tuesday, October 16, 2012 1:23 PM
> > To: user@hbase.apache.org
> > Subject: hbase can't drop a table
> >
> > I disable this table ivytest_deu , drop it .Error occurs.
> >
> >
> > ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> > missing for ivytest_deu
> > at
> > org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
> > ptor(TableEventHandler.java:174)
> > at
> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
> > TableHandler.java:44)
> > at
> > org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> > va:39)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > rImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
> > gine.java:364)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
> > 76)
> >
> > Here is some help for this command:
> > Drop the named table. Table must first be disabled. If table has
> > more than one region, run a major compaction on .META.:
> >
> > hbase> major_compact ".META."
> >
> > The major_compact ".META." doesn't work.
> > Then i try to create it ,but HBase says it .
> >
> > ERROR: Table already exists: ivytest_deu!
> >
> > After checking the region server log , the region server is always
> > trying to load this region.
> >
> > 2012-10-16 00:00:00,308 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> > open region:
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > 2012-10-16 00:00:00,313 WARN
> > org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
> > is in HBase's root directory and doesn't contain a table descriptor, do
> > consider deleting it: deu_ivytest
> > 2012-10-16 00:00:00,358 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > STARTKEY => '', ENDKEY => '', ENCODED =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,}
> > 2012-10-16 00:00:00,358 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> > handler:
> > region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
> > ationProtocol
> > 2012-10-16 00:00:00,358 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> > open of
> > region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> > starting to roll back the global memstore size.
> > 2012-10-16 00:00:00,358 INFO
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
> > of region {NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > STARTKEY => '', ENDKEY => '', ENCODED =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
> >
> > And we have a endpoint in base .After the base tried to load this table
> > ivy test_deu for 90,000 times ,the endpoint class also has been loaded
> > for 90,000 times.
> > The jvm memory has been filled.
> > The gcutil shows
> > S0C S1C S0U S1U EC EU OC OU PC
> > PU YGC YGCT FGC FGCT GCT
> > 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > 34880.0 34880.0 34648.1 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> > 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> > 34880.0 34880.0 34880.0 0.0 209472.0 209472.0 2792768.0 2792768.0
> > 71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
> >
> > The jmap dump file shows
> >
> > 3982039 instances of class org.apache.hadoop.hbase.KeyValue
> > 191050 instances of class org.apache.hadoop.fs.Path
> > 187364 instances of class
> > org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> > 187301 instances of class org.cliffc.high_scale_lib.Counter
> > 102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
> > 93652 instances of class org.apache.hadoop.hbase.HRegionInfo
> > 93650 instances of class
> > com.google.common.collect.MutableClassToInstanceMap
> > 93650 instances of class DEUColumnAggregationEndpoint
> >
> > DEUColumnAggregationEndpoint is our endpoint class.
> >
> > We guess the 90,000 times check this table and load endpoint class
> > leads this memory leak.
> >
> > But how to drop this table?
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
--
Best regards,
Ivy Tang