You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Pankaj kr <pa...@huawei.com> on 2015/11/21 11:56:15 UTC
Table regions are written in wrong path in HDFS
Hi Folks,
We met a very weird scenario.
We are running PE tool, during testing we found all regions are in transition in state FAILED_OPEN.
Region server are failed to open the regions with below exception,
2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed open of region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb., starting to roll back the global memstore size. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
java.lang.IllegalStateException: Could not instantiate a region instance.
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
... 14 more
In HDFS, Table region was written as below,
drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
-rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
Here table name "TestTable" has come twice, how is that possible?
Have anyone met this scenario?
We are using HBase 1.0.2 and HDFS 2.7.2 version.
Any suggestion/help would be much appreciated.
Regards,
Pankaj
Re: Table regions are written in wrong path in HDFS
Posted by Enis Söztutar <en...@apache.org>.
Normally this path: /hbase/data/default/TestTable/TestTable/
d5b64ad96deb3b47467db97669c009fb
should be /hbase/data/default/TestTable/d5b64ad96deb3b47467db97669c009fb.
Meaning that the table descriptor somehow was written under the wrong
folder (there should be only one TestTable in the path).
The masters logs might reveal something at the time of the table creation.
Enis
On Sun, Nov 22, 2015 at 6:02 PM, Pankaj kr <pa...@huawei.com> wrote:
> Thanks Ted,
>
> All regions are in the same condition.
> Even the table descriptor also,
>
> drwxr-xr-x - hbase hadoop 0 2015-11-18 00:04
> /hbase/data/default/TestTable/TestTable/.tabledesc
> -rw-r--r-- 3 hbase hadoop 289 2015-11-18 00:04
> /hbase/data/default/TestTable/TestTable/.tabledesc/.tableinfo.0000000001
>
> So restarting HM/RS doesn't help.
>
> Regards,
> Pankaj
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: 21 November 2015 22:24
> To: user@hbase.apache.org
> Subject: Re: Table regions are written in wrong path in HDFS
>
> Can you trace this region through master / region server log to see if
> there is some clue ?
>
> Cheers
>
> > On Nov 21, 2015, at 2:56 AM, Pankaj kr <pa...@huawei.com> wrote:
> >
> > Hi Folks,
> >
> > We met a very weird scenario.
> > We are running PE tool, during testing we found all regions are in
> transition in state FAILED_OPEN.
> >
> > Region server are failed to open the regions with below exception,
> > 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 |
> Failed open of
> region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb.,
> starting to roll back the global memstore size. |
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
> > java.lang.IllegalStateException: Could not instantiate a region instance.
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
> > at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
> > at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> > at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.reflect.InvocationTargetException
> > at
> sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
> > at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
> > ... 10 more
> > Caused by: java.lang.IllegalArgumentException: Need table descriptor
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
> > ... 14 more
> >
> > In HDFS, Table region was written as below,
> > drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05
> /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
> > -rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05
> /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
> >
> > Here table name "TestTable" has come twice, how is that possible?
> >
> > Have anyone met this scenario?
> > We are using HBase 1.0.2 and HDFS 2.7.2 version.
> >
> > Any suggestion/help would be much appreciated.
> >
> > Regards,
> > Pankaj
>
RE: Table regions are written in wrong path in HDFS
Posted by Pankaj kr <pa...@huawei.com>.
Thanks Ted,
All regions are in the same condition.
Even the table descriptor also,
drwxr-xr-x - hbase hadoop 0 2015-11-18 00:04 /hbase/data/default/TestTable/TestTable/.tabledesc
-rw-r--r-- 3 hbase hadoop 289 2015-11-18 00:04 /hbase/data/default/TestTable/TestTable/.tabledesc/.tableinfo.0000000001
So restarting HM/RS doesn't help.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: 21 November 2015 22:24
To: user@hbase.apache.org
Subject: Re: Table regions are written in wrong path in HDFS
Can you trace this region through master / region server log to see if there is some clue ?
Cheers
> On Nov 21, 2015, at 2:56 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi Folks,
>
> We met a very weird scenario.
> We are running PE tool, during testing we found all regions are in transition in state FAILED_OPEN.
>
> Region server are failed to open the regions with below exception,
> 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed open of region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb., starting to roll back the global memstore size. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Need table descriptor
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
> ... 14 more
>
> In HDFS, Table region was written as below,
> drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
> -rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
>
> Here table name "TestTable" has come twice, how is that possible?
>
> Have anyone met this scenario?
> We are using HBase 1.0.2 and HDFS 2.7.2 version.
>
> Any suggestion/help would be much appreciated.
>
> Regards,
> Pankaj
RE: Table regions are written in wrong path in HDFS
Posted by Pankaj kr <pa...@huawei.com>.
Thanks Ted,
All regions are in the same condition.
Even the table descriptor also,
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: 21 November 2015 22:24
To: user@hbase.apache.org
Subject: Re: Table regions are written in wrong path in HDFS
Can you trace this region through master / region server log to see if there is some clue ?
Cheers
> On Nov 21, 2015, at 2:56 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi Folks,
>
> We met a very weird scenario.
> We are running PE tool, during testing we found all regions are in transition in state FAILED_OPEN.
>
> Region server are failed to open the regions with below exception,
> 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed open of region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb., starting to roll back the global memstore size. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Need table descriptor
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
> ... 14 more
>
> In HDFS, Table region was written as below,
> drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
> -rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
>
> Here table name "TestTable" has come twice, how is that possible?
>
> Have anyone met this scenario?
> We are using HBase 1.0.2 and HDFS 2.7.2 version.
>
> Any suggestion/help would be much appreciated.
>
> Regards,
> Pankaj
Re: Table regions are written in wrong path in HDFS
Posted by Ted Yu <yu...@gmail.com>.
Can you trace this region through master / region server log to see if there is some clue ?
Cheers
> On Nov 21, 2015, at 2:56 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi Folks,
>
> We met a very weird scenario.
> We are running PE tool, during testing we found all regions are in transition in state FAILED_OPEN.
>
> Region server are failed to open the regions with below exception,
> 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed open of region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb., starting to roll back the global memstore size. | org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Need table descriptor
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
> ... 14 more
>
> In HDFS, Table region was written as below,
> drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
> -rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05 /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
>
> Here table name "TestTable" has come twice, how is that possible?
>
> Have anyone met this scenario?
> We are using HBase 1.0.2 and HDFS 2.7.2 version.
>
> Any suggestion/help would be much appreciated.
>
> Regards,
> Pankaj