Re: Region stuck in Transition FAILED_OPEN after upgrade to HBase0.98

[anil gupta <an...@gmail.com>]

+Phoenix Mailing List

Hi,

I found the log. This table is a catalog table of Phoenix. It seems like
this table was looking for a Phoenix class and AFAIK phoenix jars were not
present in classpath of HBase Daemons. So, it got stuck.

Here is the error log:



Failed open of region=SYSTEM.CATALOG,,1397780246020.6c12ff0021f80eea22666e4ae625b150.,
starting to roll back the global memstore size.
java.io.IOException: Unable to load configured region split policy
'org.apache.phoenix.schema.MetaDataSplitPolicy' for table
'SYSTEM.CATALOG'
	at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:121)
	at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.create(RegionSplitPolicy.java:101)
	at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:728)
	at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:685)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4479)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4449)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4421)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4377)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4328)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.phoenix.schema.MetaDataSplitPolicy
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:190)
	at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:117)
	... 14 more




How did it start working?

In meantime, I think our Hadoop Admin copied Phoenix jars in HBase
classpath dir and restarted HBase.


Thanks,
Anil Gupta


On Tue, Jul 29, 2014 at 7:55 PM, Ted Yu <yu...@gmail.com> wrote:

> bq. that the region file on HDFS is deleted by mistake
>
> Anil's case should be different since the region got finally assigned.
>
> I agree checking server log would reveal more clues.
>
> Cheers
>
>
> On Tue, Jul 29, 2014 at 7:53 PM, Qiang Tian <ti...@gmail.com> wrote:
>
> > you can search "6c12ff0021f80eea22666e4ae625b150" in regionserver log on
> > machine <host_name>.  there should be some clue why it failes to open the
> > region. I got a case recently that the region file on HDFS is deleted by
> > mistake, so the RS cannot open it.
> >
> >
> >
> > On Wed, Jul 30, 2014 at 10:45 AM, Jimmy Xiang <jx...@cloudera.com>
> wrote:
> >
> > > What's your setting hbase.assignment.timeout.management? Region is
> > > FAILED_OPEN state usually needs some manual fix, such as assign from
> > hbase
> > > shell.
> > >
> > >
> > > On Tue, Jul 29, 2014 at 6:06 PM, anil gupta <an...@gmail.com>
> > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > It seems like the problem self healed. Is there any timeout for
> > > > region_in_transition that led to this fix? Now, UI show's everything
> is
> > > > good.
> > > >
> > > > Thanks,
> > > > Anil Gupta
> > > >
> > > >
> > > > On Tue, Jul 29, 2014 at 6:01 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Can you pastebin snippet of master log pertaining to this region ?
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > On Tue, Jul 29, 2014 at 5:51 PM, anil gupta <anilgupta84@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > We recently upgrade our cluster from 0.94 to HBase0.98(cdh5.1).
> All
> > > the
> > > > > > tables are working fine except one table with 0.94.
> > > > > > One of the region has been stuck in transition since our upgrade.
> > > > > >
> > > > > > I see following HMaster UI:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 6c12ff0021f80eea22666e4ae625b150SYSTEM.CATALOG,,1397780246020.6c12ff0021f80eea22666e4ae625b150.
> > > > > > state=FAILED_OPEN, ts=Tue Jul 29 16:45:20 PDT 2014 (1469s ago),
> > > > > > server=host_name,60020,1406677501761
> > > > > >
> > > > > > How can i fix or debug this problem?
> > > > > >
> > > > > > --
> > > > > > Thanks & Regards,
> > > > > > Anil Gupta
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> >
>



-- 
Thanks & Regards,
Anil Gupta