You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Marco Gallotta <ma...@gallotta.co.za> on 2012/08/10 23:22:07 UTC
Table listed in "list", but not in .META.
Hi there
I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
Here are some logs from master (foo is one of the tables not found):
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:
2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
Any ideas how to find and remove any references to these non-existent tables?
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
Hi Marco,
There seems to be some problem with the regionserver hosting the
table. Had you disabled or deleted "foo"?And try to see what "hbck"
says. And RegionAlreadyInTransitionException: is normally thrown if a
region server is asked to open or close a region but it's already
processing that region. BTW, did you find anything abnormal with your
HDFS??
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 2:52 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
In core-site.xml:
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop/${user.name}</value>
</property>
In hdfs-site.xml:
<property>
<name>dfs.name.dir</name>
<value>/var/hdfs-data-name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/var/hdfs-data</value>
</property>
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:34 PM, Mohammad Tariq wrote:
> Have you specified "hadoop.tmp.dir" property in your core-site.xml and
> "dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
> files??
> If not you will loose all your data along with you meta information as
> Anil has said.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <anilgupta84@gmail.com (mailto:anilgupta84@gmail.com)> wrote:
> > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > rebooted your machined then you will have problems.
> >
> > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> >
> > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > gathering more data.
> > >
> > > I get the following error when running hbck -repair, and then it stalls:
> > >
> > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > after fix...
> > > Version: 0.94.0
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > HDFS...
> > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > >
> > >
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > >
> > > > Is it a standalone installation or pseudo-distributed?
> > > > I faced a similar problem a few days back in a distributed cluster and
> > > >
> > >
> > > used
> > > > hbck -repair option. You might give it a try.
> > > >
> > > > ~Anil
> > > >
> > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > >
> > > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > > compaction and see if ti works.
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > wrote:
> > > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > > >
> > > > >
> > > >
> > >
> > > So
> > > > >
> > > > > the reference to localhost is correct.
> > > > > >
> > > > > > --
> > > > > > Marco Gallotta | Mountain View, California
> > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > >
> > > >
> > >
> > > 417-3313
> > > > > >
> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > >
> > > > > >
> > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > >
> > > > > > > Are you running a distributed cluster?
> > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > >
> > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > region
> > > > > > > server
> > > > > > >
> > > > > > > ~Anil
> > > > > > >
> > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > >
> > > > > > > > Here's the output from hbck -details:
> > > http://pastebin.com/ZxVZEctY
> > > > > > > >
> > > > > > > > Extract:
> > > > > > > >
> > > > > > > > 6 inconsistencies detected.
> > > > > > > > Status: INCONSISTENT
> > > > > > > >
> > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > operated on
> > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > both ops
> > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > in a
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > table
> > > > > > > > list, although I did create it at one point but was able to
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > clear it
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > from
> > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > table
> > > > > > > > creation when I was trying to get lzo compression working and
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > table
> > > > > > > > creation was hanging.
> > > > > > > >
> > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > but I
> > > > > want
> > > > > > > > to proceed with caution.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > 417-3313
> > > > > > > >
> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > >
> > > > > > > > > Hi Marco,
> > > > > > > > >
> > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > >
> > > > > > > > > ~Anil
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > >
> > > > > > > > > > Hi there
> > > > > > > > > >
> > > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > > but
> > > > > produce
> > > > > > > > > > "table not found" when performing any operation on them.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > There is
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > no
> > > > > > > > > > reference of them in the .META. table. It seems to be
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > resulting in
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > some of
> > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > >
> > > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > > found):
> > > > > > > > > >
> > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > Unexpected state :
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ..
> > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > There are also a number of the following types of error logs:
> > > > > > > > > >
> > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > assignment in:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > due to
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > Received:OPEN for the
> > > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > ,which we
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > are
> > > > > > > > > > already trying to OPEN.
> > > > > > > > > >
> > > > > > > > > > Any ideas how to find and remove any references to these
> > > > > non-existent
> > > > > > > > > > tables?
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > (650)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > 417-3313
> > > > > > > > > >
> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Anil Gupta
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Thanks & Regards,
> > > > > > > Anil Gupta
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
Have you specified "hadoop.tmp.dir" property in your core-site.xml and
"dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
files??
If not you will loose all your data along with you meta information as
Anil has said.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <an...@gmail.com> wrote:
> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> rebooted your machined then you will have problems.
>
> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
>
>> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> gathering more data.
>>
>> I get the following error when running hbck -repair, and then it stalls:
>>
>> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> after fix...
>> Version: 0.94.0
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> HDFS...
>> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> at
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>
>>
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> marco@gallotta.co.za | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>
>> > Is it a standalone installation or pseudo-distributed?
>> > I faced a similar problem a few days back in a distributed cluster and
>> used
>> > hbck -repair option. You might give it a try.
>> >
>> > ~Anil
>> >
>> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com(mailto:
>> dontariq@gmail.com)> wrote:
>> >
>> > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > compaction and see if ti works.
>> > >
>> > > Regards,
>> > > Mohammad Tariq
>> > >
>> > >
>> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za(mailto:
>> marco@gallotta.co.za)>
>> > > wrote:
>> > > > It's not a distributed cluster. I'm not processing enough data yet.
>> So
>> > >
>> > > the reference to localhost is correct.
>> > > >
>> > > > --
>> > > > Marco Gallotta | Mountain View, California
>> > > > Software Engineer, Infrastructure | Loki Studios
>> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> twitter.com/marcog (http://twitter.com/marcog)
>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> 417-3313
>> > > >
>> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > >
>> > > >
>> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > >
>> > > > > Are you running a distributed cluster?
>> > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > >
>> > > > > You are getting reference to localhost in hbck output:
>> > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> region
>> > > > > server
>> > > > >
>> > > > > ~Anil
>> > > > >
>> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > >
>> > > > > > Here's the output from hbck -details:
>> http://pastebin.com/ZxVZEctY
>> > > > > >
>> > > > > > Extract:
>> > > > > >
>> > > > > > 6 inconsistencies detected.
>> > > > > > Status: INCONSISTENT
>> > > > > >
>> > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > operated on
>> > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > >
>> > > >
>> > >
>> > > both ops
>> > > > > > say table not found). I also just noticed "foo" does not occur
>> in a
>> > > > >
>> > > >
>> > >
>> > > table
>> > > > > > list, although I did create it at one point but was able to
>> clear it
>> > > > >
>> > > >
>> > >
>> > > from
>> > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> table
>> > > > > > creation when I was trying to get lzo compression working and
>> table
>> > > > > > creation was hanging.
>> > > > > >
>> > > > > > Is there any way to repair this? I see hbck has repair options,
>> but I
>> > > want
>> > > > > > to proceed with caution.
>> > > > > >
>> > > > > > --
>> > > > > > Marco Gallotta | Mountain View, California
>> > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > >
>> > > >
>> > >
>> > > 417-3313
>> > > > > >
>> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > >
>> > > > > >
>> > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > >
>> > > > > > > Hi Marco,
>> > > > > > >
>> > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > >
>> > > > > > > ~Anil
>> > > > > > >
>> > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > >
>> > > > > > > > Hi there
>> > > > > > > >
>> > > > > > > > I have a few tables which show up in a "list" in the shell,
>> but
>> > > produce
>> > > > > > > > "table not found" when performing any operation on them.
>> There is
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > no
>> > > > > > > > reference of them in the .META. table. It seems to be
>> resulting in
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > some of
>> > > > > > > > the hbase services being killed every so often.
>> > > > > > > >
>> > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > found):
>> > > > > > > >
>> > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > Master server abort: loaded coprocessors are: []
>> > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > Unexpected state :
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > ..
>> > > > > > > > Cannot transit it to OFFLINE.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > There are also a number of the following types of error logs:
>> > > > > > > >
>> > > > > > > > 2012-08-09 20:10:04,308 ERROR
>> > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > assignment in:
>> > > > > > > >
>> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > due to
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > > > > > > Received:OPEN for the
>> > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > ,which we
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > are
>> > > > > > > > already trying to OPEN.
>> > > > > > > >
>> > > > > > > > Any ideas how to find and remove any references to these
>> > > non-existent
>> > > > > > > > tables?
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>> (650)
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > 417-3313
>> > > > > > > >
>> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Thanks & Regards,
>> > > > > > > Anil Gupta
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Thanks & Regards,
>> > > > > Anil Gupta
>> > > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Anil Gupta
>> >
>> >
>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
Same exception on repair after reboot. :/
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:09 PM, Anil Gupta wrote:
> Can you try to reboot the machine and run repair again. Might not sound logical but I would give it a shot.
>
> PS: My personal experience is that hbase and hadoop has never been reliable in my standalone environment. I always trust the distributed cluster environment. AFAIK, these things are tested extensively in distributed mode.
>
> Best Regards,
> Anil
>
> On Aug 10, 2012, at 5:00 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
>
> > Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
> >
> > Configured Capacity: 211474616320 (196.95 GB)
> > Present Capacity: 169354764288 (157.72 GB)
> > DFS Remaining: 162401554432 (151.25 GB)
> > DFS Used: 6953209856 (6.48 GB)
> > DFS Used%: 4.11%
> > Under replicated blocks: 436
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 1 (1 total, 0 dead)
> >
> > Name: 127.0.0.1:50010
> > Decommission Status : Normal
> > Configured Capacity: 211474616320 (196.95 GB)
> > DFS Used: 6953209856 (6.48 GB)
> > Non DFS Used: 42119852032 (39.23 GB)
> > DFS Remaining: 162401554432(151.25 GB)
> > DFS Used%: 3.29%
> > DFS Remaining%: 76.79%
> > Last contact: Fri Aug 10 16:59:26 PDT 2012
> >
> >
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
> >
> > > If your HDFS is in safemode you'll get something like this :
> > >
> > > cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> > > Safe mode is ON
> > > Configured Capacity: 31111143424 (28.97 GB)
> > > Present Capacity: 5309755392 (4.95 GB)
> > > DFS Remaining: 4799320064 (4.47 GB)
> > > DFS Used: 510435328 (486.79 MB)
> > > DFS Used%: 9.61%
> > > Under replicated blocks: 1
> > > Blocks with corrupt replicas: 0
> > > Missing blocks: 0
> > >
> > > -------------------------------------------------
> > > Datanodes available: 1 (1 total, 0 dead)
> > >
> > > Name: 127.0.0.1:50010
> > > Decommission Status : Normal
> > > Configured Capacity: 31111143424 (28.97 GB)
> > > DFS Used: 510435328 (486.79 MB)
> > > Non DFS Used: 25801388032 (24.03 GB)
> > > DFS Remaining: 4799320064(4.47 GB)
> > > DFS Used%: 1.64%
> > > DFS Remaining%: 15.43%
> > > Last contact: Sat Aug 11 05:19:18 IST 2012
> > >
> > > See the line in red color.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
> > > > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > > > point your web browser to http://localhost:9000. It'll show all the
> > > > details of your HDFS.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
> > > wrote:
> > > > > How do you check that?
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > > > >
> > > > > > This is pretty strange. I mean everything seems to be in place, but we
> > > > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > > > >
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > > > What about fs.default.name (http://fs.default.name)?????
> > > > > > >
> > > > > > > Regards,
> > > > > > > Mohammad Tariq
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))> wrote:
> > > > > > > > It's in /var which is persistent across reboots.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > 417-3313
> > > > > > > >
> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > > > > > > >
> > > > > > > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp
> > > and you have
> > > > > > > > > rebooted your machined then you will have problems.
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > > > > > > > >
> > > > > > > > > > It's a pseudo-distributed cluster, as I plan to add more nodes
> > > as we start
> > > > > > > > > > gathering more data.
> > > > > > > > > >
> > > > > > > > > > I get the following error when running hbck -repair, and then
> > > it stalls:
> > > > > > > > > >
> > > > > > > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before
> > > re-checking
> > > > > > > > > > after fix...
> > > > > > > > > > Version: 0.94.0
> > > > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > from
> > > > > > > > > > HDFS...
> > > > > > > > > > Exception in thread "main"
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > java.util.concurrent.RejectedExecutionException
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > 417-3313
> > > > > > > > > >
> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > > > > > > >
> > > > > > > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > > > > > > I faced a similar problem a few days back in a distributed
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > cluster and
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > used
> > > > > > > > > > > hbck -repair option. You might give it a try.
> > > > > > > > > > >
> > > > > > > > > > > ~Anil
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
> > > dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > > > > > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Could you please share your /etc/hosts file??Meantime, do a
> > > manual
> > > > > > > > > > > > compaction and see if ti works.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Mohammad Tariq
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > It's not a distributed cluster. I'm not processing enough
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > data yet.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > So
> > > > > > > > > > > >
> > > > > > > > > > > > the reference to localhost is correct.
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > (650)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 417-3313
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Are you running a distributed cluster?
> > > > > > > > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > > > > > > > deployed => } on HDFS, but not listed in META or
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > deployed on any
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > region
> > > > > > > > > > > > > > server
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ~Anil
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's the output from hbck -details:
> > > > > > > > > > http://pastebin.com/ZxVZEctY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Extract:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6 inconsistencies detected.
> > > > > > > > > > > > > > > Status: INCONSISTENT
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6 is the number of tables that appear in "list" but
> > > cannot be
> > > > > > > > > > > > operated on
> > > > > > > > > > > > > > > (which btw, includes not being able to run
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > disable/drop on them -
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > both ops
> > > > > > > > > > > > > > > say table not found). I also just noticed "foo" does
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > not occur
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > in a
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > table
> > > > > > > > > > > > > > > list, although I did create it at one point but was
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > able to
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > clear it
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > .META. when it also was reporting table not found
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > when trying to
> > > > > > > > > > > > > > > disable/drop it. All these come from when I ^C'ed
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > (i.e. killed)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > table
> > > > > > > > > > > > > > > creation when I was trying to get lzo compression
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > working and
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > table
> > > > > > > > > > > > > > > creation was hanging.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is there any way to repair this? I see hbck has
> > > repair options,
> > > > > > > > > > but I
> > > > > > > > > > > > want
> > > > > > > > > > > > > > > to proceed with caution.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) |
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > +1 (650)
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 417-3313
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Marco,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > > > > > > > Run: 'hbase hbck -help' to get all the available
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > options.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ~Anil
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > ))>wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have a few tables which show up in a "list" in
> > > the shell,
> > > > > > > > > > but
> > > > > > > > > > > > produce
> > > > > > > > > > > > > > > > > "table not found" when performing any operation
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > on them.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > There is
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > reference of them in the .META. table. It seems
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > to be
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > resulting in
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Here are some logs from master (foo is one of the
> > > tables not
> > > > > > > > > > > > found):
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > > > Unexpected state :
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ..
> > > > > > > > > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are also a number of the following types of
> > > error logs:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > Failed
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > assignment in:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > due to
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > > > > > > > > Received:OPEN for the
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > ,which we
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > already trying to OPEN.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Any ideas how to find and remove any references
> > > to these
> > > > > > > > > > > > non-existent
> > > > > > > > > > > > > > > > > tables?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > |
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > | +1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > (650)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 417-3313
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Sent with Sparrow (
> > > http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > Anil Gupta
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Anil Gupta
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Anil Gupta <an...@gmail.com>.
Can you try to reboot the machine and run repair again. Might not sound logical but I would give it a shot.
PS: My personal experience is that hbase and hadoop has never been reliable in my standalone environment. I always trust the distributed cluster environment. AFAIK, these things are tested extensively in distributed mode.
Best Regards,
Anil
On Aug 10, 2012, at 5:00 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
>
> Configured Capacity: 211474616320 (196.95 GB)
> Present Capacity: 169354764288 (157.72 GB)
> DFS Remaining: 162401554432 (151.25 GB)
> DFS Used: 6953209856 (6.48 GB)
> DFS Used%: 4.11%
> Under replicated blocks: 436
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 211474616320 (196.95 GB)
> DFS Used: 6953209856 (6.48 GB)
> Non DFS Used: 42119852032 (39.23 GB)
> DFS Remaining: 162401554432(151.25 GB)
> DFS Used%: 3.29%
> DFS Remaining%: 76.79%
> Last contact: Fri Aug 10 16:59:26 PDT 2012
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
>
>> If your HDFS is in safemode you'll get something like this :
>>
>> cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
>> Safe mode is ON
>> Configured Capacity: 31111143424 (28.97 GB)
>> Present Capacity: 5309755392 (4.95 GB)
>> DFS Remaining: 4799320064 (4.47 GB)
>> DFS Used: 510435328 (486.79 MB)
>> DFS Used%: 9.61%
>> Under replicated blocks: 1
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 1 (1 total, 0 dead)
>>
>> Name: 127.0.0.1:50010
>> Decommission Status : Normal
>> Configured Capacity: 31111143424 (28.97 GB)
>> DFS Used: 510435328 (486.79 MB)
>> Non DFS Used: 25801388032 (24.03 GB)
>> DFS Remaining: 4799320064(4.47 GB)
>> DFS Used%: 1.64%
>> DFS Remaining%: 15.43%
>> Last contact: Sat Aug 11 05:19:18 IST 2012
>>
>> See the line in red color.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
>>> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
>>> point your web browser to http://localhost:9000. It'll show all the
>>> details of your HDFS.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
>> wrote:
>>>> How do you check that?
>>>>
>>>> --
>>>> Marco Gallotta | Mountain View, California
>>>> Software Engineer, Infrastructure | Loki Studios
>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>>>>
>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>
>>>>
>>>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>>>
>>>>> This is pretty strange. I mean everything seems to be in place, but we
>>>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>>>
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>>
>>>>>
>>>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>> dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>>>>> What about fs.default.name (http://fs.default.name)?????
>>>>>>
>>>>>> Regards,
>>>>>> Mohammad Tariq
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))> wrote:
>>>>>>> It's in /var which is persistent across reboots.
>>>>>>>
>>>>>>> --
>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>>>>>>
>>>>>
>>>>
>>>
>>
>> 417-3313
>>>>>>>
>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>
>>>>>>>
>>>>>>> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>>>>>>
>>>>>>>> Where are you storing your hdfs data? Is it /tmp? If it's /tmp
>> and you have
>>>>>>>> rebooted your machined then you will have problems.
>>>>>>>>
>>>>>>>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>>>>>>>
>>>>>>>>> It's a pseudo-distributed cluster, as I plan to add more nodes
>> as we start
>>>>>>>>> gathering more data.
>>>>>>>>>
>>>>>>>>> I get the following error when running hbck -repair, and then
>> it stalls:
>>>>>>>>>
>>>>>>>>> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before
>> re-checking
>>>>>>>>> after fix...
>>>>>>>>> Version: 0.94.0
>>>>>>>>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>>>>>>>>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> from
>>>>>>>>> HDFS...
>>>>>>>>> Exception in thread "main"
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> java.util.concurrent.RejectedExecutionException
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>>>>>>>>> at
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> 417-3313
>>>>>>>>>
>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>>>>>>>>
>>>>>>>>>> Is it a standalone installation or pseudo-distributed?
>>>>>>>>>> I faced a similar problem a few days back in a distributed
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> cluster and
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> used
>>>>>>>>>> hbck -repair option. You might give it a try.
>>>>>>>>>>
>>>>>>>>>> ~Anil
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
>> dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>>>>>>>>> dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>>>>>>>>>
>>>>>>>>>>> Could you please share your /etc/hosts file??Meantime, do a
>> manual
>>>>>>>>>>> compaction and see if ti works.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Mohammad Tariq
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> It's not a distributed cluster. I'm not processing enough
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> data yet.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So
>>>>>>>>>>>
>>>>>>>>>>> the reference to localhost is correct.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> (650)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 417-3313
>>>>>>>>>>>>
>>>>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Are you running a distributed cluster?
>>>>>>>>>>>>> If yes, do you have localhost in /etc/hosts file?
>>>>>>>>>>>>>
>>>>>>>>>>>>> You are getting reference to localhost in hbck output:
>>>>>>>>>>>>> ERROR: Region { meta => null, hdfs =>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>>>>>>>>>>>>> deployed => } on HDFS, but not listed in META or
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> deployed on any
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> region
>>>>>>>>>>>>> server
>>>>>>>>>>>>>
>>>>>>>>>>>>> ~Anil
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the output from hbck -details:
>>>>>>>>> http://pastebin.com/ZxVZEctY
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Extract:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 6 inconsistencies detected.
>>>>>>>>>>>>>> Status: INCONSISTENT
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 6 is the number of tables that appear in "list" but
>> cannot be
>>>>>>>>>>> operated on
>>>>>>>>>>>>>> (which btw, includes not being able to run
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> disable/drop on them -
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> both ops
>>>>>>>>>>>>>> say table not found). I also just noticed "foo" does
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> not occur
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> in a
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> table
>>>>>>>>>>>>>> list, although I did create it at one point but was
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> able to
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> clear it
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> from
>>>>>>>>>>>>>> .META. when it also was reporting table not found
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> when trying to
>>>>>>>>>>>>>> disable/drop it. All these come from when I ^C'ed
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> (i.e. killed)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> table
>>>>>>>>>>>>>> creation when I was trying to get lzo compression
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> working and
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> table
>>>>>>>>>>>>>> creation was hanging.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any way to repair this? I see hbck has
>> repair options,
>>>>>>>>> but I
>>>>>>>>>>> want
>>>>>>>>>>>>>> to proceed with caution.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) |
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> +1 (650)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 417-3313
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Marco,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Did anything disastrous happen to cluster?
>>>>>>>>>>>>>>> Can you try using hbck utility of HBase.
>>>>>>>>>>>>>>> Run: 'hbase hbck -help' to get all the available
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> options.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ~Anil
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> ))>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi there
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have a few tables which show up in a "list" in
>> the shell,
>>>>>>>>> but
>>>>>>>>>>> produce
>>>>>>>>>>>>>>>> "table not found" when performing any operation
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> on them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> no
>>>>>>>>>>>>>>>> reference of them in the .META. table. It seems
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> to be
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> resulting in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> some of
>>>>>>>>>>>>>>>> the hbase services being killed every so often.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here are some logs from master (foo is one of the
>> tables not
>>>>>>>>>>> found):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2012-08-09 20:40:44,301 FATAL
>>>>>>>>>>> org.apache.hadoop.hbase.master.HMaster:
>>>>>>>>>>>>>>>> Master server abort: loaded coprocessors are: []
>>>>>>>>>>>>>>>> 2012-08-09 20:40:44,301 FATAL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.hadoop.hbase.master.HMaster:
>>>>>>>>>>>>>>>> Unexpected state :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>>>>>>>>>>>>>>> state=PENDING_OPEN, ts=1344570044277,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ..
>>>>>>>>>>>>>>>> Cannot transit it to OFFLINE.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There are also a number of the following types of
>> error logs:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2012-08-09 20:10:04,308 ERROR
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.master.AssignmentManager:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> Failed
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> assignment in:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> due to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>>>>>>>>>>>>>>>> Received:OPEN for the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ,which we
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> already trying to OPEN.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any ideas how to find and remove any references
>> to these
>>>>>>>>>>> non-existent
>>>>>>>>>>>>>>>> tables?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> |
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> | +1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (650)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 417-3313
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sent with Sparrow (
>> http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>> Anil Gupta
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Anil Gupta
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Anil Gupta
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks & Regards,
>>>>>>>> Anil Gupta
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
Yeah, I feel the same. I am still in the learning phase, so it is quite
possible that I might be missing something important. But we have several
experts on the list and I hope they post their response for you.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:30 AM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help
> so far though!
>
> Configured Capacity: 211474616320 (196.95 GB)
> Present Capacity: 169354764288 (157.72 GB)
> DFS Remaining: 162401554432 (151.25 GB)
> DFS Used: 6953209856 (6.48 GB)
> DFS Used%: 4.11%
> Under replicated blocks: 436
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 211474616320 (196.95 GB)
> DFS Used: 6953209856 (6.48 GB)
> Non DFS Used: 42119852032 (39.23 GB)
> DFS Remaining: 162401554432(151.25 GB)
> DFS Used%: 3.29%
> DFS Remaining%: 76.79%
> Last contact: Fri Aug 10 16:59:26 PDT 2012
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
>
> > If your HDFS is in safemode you'll get something like this :
> >
> > cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> > Safe mode is ON
> > Configured Capacity: 31111143424 (28.97 GB)
> > Present Capacity: 5309755392 (4.95 GB)
> > DFS Remaining: 4799320064 (4.47 GB)
> > DFS Used: 510435328 (486.79 MB)
> > DFS Used%: 9.61%
> > Under replicated blocks: 1
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 1 (1 total, 0 dead)
> >
> > Name: 127.0.0.1:50010
> > Decommission Status : Normal
> > Configured Capacity: 31111143424 (28.97 GB)
> > DFS Used: 510435328 (486.79 MB)
> > Non DFS Used: 25801388032 (24.03 GB)
> > DFS Remaining: 4799320064(4.47 GB)
> > DFS Used%: 1.64%
> > DFS Remaining%: 15.43%
> > Last contact: Sat Aug 11 05:19:18 IST 2012
> >
> > See the line in red color.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <dontariq@gmail.com(mailto:
> dontariq@gmail.com)> wrote:
> > > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > > point your web browser to http://localhost:9000. It'll show all the
> > > details of your HDFS.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za(mailto:
> marco@gallotta.co.za)>
> > wrote:
> > > > How do you check that?
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > > >
> > > > > This is pretty strange. I mean everything seems to be in place,
> but we
> > > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <
> dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > > What about fs.default.name (http://fs.default.name)?????
> > > > > >
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <
> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))> wrote:
> > > > > > > It's in /var which is persistent across reboots.
> > > > > > >
> > > > > > > --
> > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > >
> > > > >
> > > >
> > >
> >
> > 417-3313
> > > > > > >
> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > >
> > > > > > >
> > > > > > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > > > > > >
> > > > > > > > Where are you storing your hdfs data? Is it /tmp? If it's
> /tmp
> > and you have
> > > > > > > > rebooted your machined then you will have problems.
> > > > > > > >
> > > > > > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > > > > > > >
> > > > > > > > > It's a pseudo-distributed cluster, as I plan to add more
> nodes
> > as we start
> > > > > > > > > gathering more data.
> > > > > > > > >
> > > > > > > > > I get the following error when running hbck -repair, and
> then
> > it stalls:
> > > > > > > > >
> > > > > > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms
> before
> > re-checking
> > > > > > > > > after fix...
> > > > > > > > > Version: 0.94.0
> > > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos
> HDFS
> > > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase
> regioninfo
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > from
> > > > > > > > > HDFS...
> > > > > > > > > Exception in thread "main"
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > java.util.concurrent.RejectedExecutionException
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> (650)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > 417-3313
> > > > > > > > >
> > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > > > > > >
> > > > > > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > > > > > I faced a similar problem a few days back in a
> distributed
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > cluster and
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > used
> > > > > > > > > > hbck -repair option. You might give it a try.
> > > > > > > > > >
> > > > > > > > > > ~Anil
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
> > dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > > > > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > > > > > >
> > > > > > > > > > > Could you please share your /etc/hosts file??Meantime,
> do a
> > manual
> > > > > > > > > > > compaction and see if ti works.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Mohammad Tariq
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > It's not a distributed cluster. I'm not processing
> enough
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > data yet.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > So
> > > > > > > > > > >
> > > > > > > > > > > the reference to localhost is correct.
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)
> | +1
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > (650)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 417-3313
> > > > > > > > > > > >
> > > > > > > > > > > > Sent with Sparrow (
> http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta
> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Are you running a distributed cluster?
> > > > > > > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > > > > > >
> > > > > > > > > > > > > You are getting reference to localhost in hbck
> output:
> > > > > > > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > > > > > > deployed => } on HDFS, but not listed in META or
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > deployed on any
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > region
> > > > > > > > > > > > > server
> > > > > > > > > > > > >
> > > > > > > > > > > > > ~Anil
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za
> ))>wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's the output from hbck -details:
> > > > > > > > > http://pastebin.com/ZxVZEctY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Extract:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 6 inconsistencies detected.
> > > > > > > > > > > > > > Status: INCONSISTENT
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 6 is the number of tables that appear in "list"
> but
> > cannot be
> > > > > > > > > > > operated on
> > > > > > > > > > > > > > (which btw, includes not being able to run
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > disable/drop on them -
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > both ops
> > > > > > > > > > > > > > say table not found). I also just noticed "foo"
> does
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > not occur
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > in a
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > table
> > > > > > > > > > > > > > list, although I did create it at one point but
> was
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > able to
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > clear it
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > from
> > > > > > > > > > > > > > .META. when it also was reporting table not found
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > when trying to
> > > > > > > > > > > > > > disable/drop it. All these come from when I ^C'ed
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > (i.e. killed)
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > table
> > > > > > > > > > > > > > creation when I was trying to get lzo compression
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > working and
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > table
> > > > > > > > > > > > > > creation was hanging.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is there any way to repair this? I see hbck has
> > repair options,
> > > > > > > > > but I
> > > > > > > > > > > want
> > > > > > > > > > > > > > to proceed with caution.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > > fb.me/marco.gallotta (
> http://fb.me/marco.gallotta) |
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > > marco@gallotta.co.za (mailto:
> marco@gallotta.co.za) |
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > +1 (650)
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 417-3313
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sent with Sparrow (
> http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta
> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Marco,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > > > > > > Run: 'hbase hbck -help' to get all the
> available
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > options.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ~Anil
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco
> Gallotta <
> > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za
> )(mailto:
> > > > > > > > > > > > > > marco@gallotta.co.za (mailto:
> marco@gallotta.co.za
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ))>wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have a few tables which show up in a
> "list" in
> > the shell,
> > > > > > > > > but
> > > > > > > > > > > produce
> > > > > > > > > > > > > > > > "table not found" when performing any
> operation
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > on them.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There is
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > no
> > > > > > > > > > > > > > > > reference of them in the .META. table. It
> seems
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > to be
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > resulting in
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > > the hbase services being killed every so
> often.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here are some logs from master (foo is one
> of the
> > tables not
> > > > > > > > > > > found):
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > > Master server abort: loaded coprocessors
> are: []
> > > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > > Unexpected state :
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ..
> > > > > > > > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are also a number of the following
> types of
> > error logs:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > > > > > > >
> org.apache.hadoop.hbase.master.AssignmentManager:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > Failed
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > assignment in:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > due to
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > > > > > > > Received:OPEN for the
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ,which we
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > already trying to OPEN.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Any ideas how to find and remove any
> references
> > to these
> > > > > > > > > > > non-existent
> > > > > > > > > > > > > > > > tables?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > > > > Software Engineer, Infrastructure | Loki
> Studios
> > > > > > > > > > > > > > > > fb.me/marco.gallotta (
> http://fb.me/marco.gallotta)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > |
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > > > > marco@gallotta.co.za (mailto:
> marco@gallotta.co.za)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > | +1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > (650)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 417-3313
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sent with Sparrow (
> > http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Thanks & Regards,
> > > > > > > > > > Anil Gupta
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks & Regards,
> > > > > > > > Anil Gupta
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
Configured Capacity: 211474616320 (196.95 GB)
Present Capacity: 169354764288 (157.72 GB)
DFS Remaining: 162401554432 (151.25 GB)
DFS Used: 6953209856 (6.48 GB)
DFS Used%: 4.11%
Under replicated blocks: 436
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 211474616320 (196.95 GB)
DFS Used: 6953209856 (6.48 GB)
Non DFS Used: 42119852032 (39.23 GB)
DFS Remaining: 162401554432(151.25 GB)
DFS Used%: 3.29%
DFS Remaining%: 76.79%
Last contact: Fri Aug 10 16:59:26 PDT 2012
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
> If your HDFS is in safemode you'll get something like this :
>
> cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
> Safe mode is ON
> Configured Capacity: 31111143424 (28.97 GB)
> Present Capacity: 5309755392 (4.95 GB)
> DFS Remaining: 4799320064 (4.47 GB)
> DFS Used: 510435328 (486.79 MB)
> DFS Used%: 9.61%
> Under replicated blocks: 1
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 31111143424 (28.97 GB)
> DFS Used: 510435328 (486.79 MB)
> Non DFS Used: 25801388032 (24.03 GB)
> DFS Remaining: 4799320064(4.47 GB)
> DFS Used%: 1.64%
> DFS Remaining%: 15.43%
> Last contact: Sat Aug 11 05:19:18 IST 2012
>
> See the line in red color.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
> > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > point your web browser to http://localhost:9000. It'll show all the
> > details of your HDFS.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
> wrote:
> > > How do you check that?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > >
> > > > This is pretty strange. I mean everything seems to be in place, but we
> > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > What about fs.default.name (http://fs.default.name)?????
> > > > >
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > >
> > > > >
> > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> marco@gallotta.co.za (mailto:marco@gallotta.co.za))> wrote:
> > > > > > It's in /var which is persistent across reboots.
> > > > > >
> > > > > > --
> > > > > > Marco Gallotta | Mountain View, California
> > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > >
> > > > >
> > > >
> > >
> >
>
> twitter.com/marcog (http://twitter.com/marcog)
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > >
> > > >
> > >
> >
>
> 417-3313
> > > > > >
> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > >
> > > > > >
> > > > > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > > > > >
> > > > > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp
> and you have
> > > > > > > rebooted your machined then you will have problems.
> > > > > > >
> > > > > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
> marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > > > > > >
> > > > > > > > It's a pseudo-distributed cluster, as I plan to add more nodes
> as we start
> > > > > > > > gathering more data.
> > > > > > > >
> > > > > > > > I get the following error when running hbck -repair, and then
> it stalls:
> > > > > > > >
> > > > > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before
> re-checking
> > > > > > > > after fix...
> > > > > > > > Version: 0.94.0
> > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> from
> > > > > > > > HDFS...
> > > > > > > > Exception in thread "main"
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> java.util.concurrent.RejectedExecutionException
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> 417-3313
> > > > > > > >
> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > > > > >
> > > > > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > > > > I faced a similar problem a few days back in a distributed
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> cluster and
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > used
> > > > > > > > > hbck -repair option. You might give it a try.
> > > > > > > > >
> > > > > > > > > ~Anil
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
> dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > > > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > > > > >
> > > > > > > > > > Could you please share your /etc/hosts file??Meantime, do a
> manual
> > > > > > > > > > compaction and see if ti works.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Mohammad Tariq
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > > > > > > wrote:
> > > > > > > > > > > It's not a distributed cluster. I'm not processing enough
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> data yet.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > So
> > > > > > > > > >
> > > > > > > > > > the reference to localhost is correct.
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> (650)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 417-3313
> > > > > > > > > > >
> > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Are you running a distributed cluster?
> > > > > > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > > > > >
> > > > > > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > > > > > deployed => } on HDFS, but not listed in META or
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> deployed on any
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > region
> > > > > > > > > > > > server
> > > > > > > > > > > >
> > > > > > > > > > > > ~Anil
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Here's the output from hbck -details:
> > > > > > > > http://pastebin.com/ZxVZEctY
> > > > > > > > > > > > >
> > > > > > > > > > > > > Extract:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6 inconsistencies detected.
> > > > > > > > > > > > > Status: INCONSISTENT
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6 is the number of tables that appear in "list" but
> cannot be
> > > > > > > > > > operated on
> > > > > > > > > > > > > (which btw, includes not being able to run
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> disable/drop on them -
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > both ops
> > > > > > > > > > > > > say table not found). I also just noticed "foo" does
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> not occur
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > in a
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > table
> > > > > > > > > > > > > list, although I did create it at one point but was
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> able to
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > clear it
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > from
> > > > > > > > > > > > > .META. when it also was reporting table not found
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> when trying to
> > > > > > > > > > > > > disable/drop it. All these come from when I ^C'ed
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> (i.e. killed)
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > table
> > > > > > > > > > > > > creation when I was trying to get lzo compression
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> working and
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > table
> > > > > > > > > > > > > creation was hanging.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Is there any way to repair this? I see hbck has
> repair options,
> > > > > > > > but I
> > > > > > > > > > want
> > > > > > > > > > > > > to proceed with caution.
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) |
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> +1 (650)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 417-3313
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Marco,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > > > > > Run: 'hbase hbck -help' to get all the available
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> options.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ~Anil
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ))>wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have a few tables which show up in a "list" in
> the shell,
> > > > > > > > but
> > > > > > > > > > produce
> > > > > > > > > > > > > > > "table not found" when performing any operation
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> on them.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > There is
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > no
> > > > > > > > > > > > > > > reference of them in the .META. table. It seems
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> to be
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > resulting in
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > some of
> > > > > > > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here are some logs from master (foo is one of the
> tables not
> > > > > > > > > > found):
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > > > > Unexpected state :
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > ..
> > > > > > > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are also a number of the following types of
> error logs:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> Failed
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > assignment in:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > due to
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > > > > > > Received:OPEN for the
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ,which we
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > already trying to OPEN.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any ideas how to find and remove any references
> to these
> > > > > > > > > > non-existent
> > > > > > > > > > > > > > > tables?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> |
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> | +1
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > (650)
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 417-3313
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sent with Sparrow (
> http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > Anil Gupta
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Anil Gupta
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Thanks & Regards,
> > > > > > > Anil Gupta
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
If your HDFS is in safemode you'll get something like this :
cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
Safe mode is ON
Configured Capacity: 31111143424 (28.97 GB)
Present Capacity: 5309755392 (4.95 GB)
DFS Remaining: 4799320064 (4.47 GB)
DFS Used: 510435328 (486.79 MB)
DFS Used%: 9.61%
Under replicated blocks: 1
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 31111143424 (28.97 GB)
DFS Used: 510435328 (486.79 MB)
Non DFS Used: 25801388032 (24.03 GB)
DFS Remaining: 4799320064(4.47 GB)
DFS Used%: 1.64%
DFS Remaining%: 15.43%
Last contact: Sat Aug 11 05:19:18 IST 2012
See the line in red color.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <do...@gmail.com> wrote:
> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> point your web browser to http://localhost:9000. It'll show all the
> details of your HDFS.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za>
wrote:
>> How do you check that?
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> marco@gallotta.co.za | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>
>>> This is pretty strange. I mean everything seems to be in place, but we
>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com(mailto:
dontariq@gmail.com)> wrote:
>>> > What about fs.default.name (http://fs.default.name)?????
>>> >
>>> > Regards,
>>> > Mohammad Tariq
>>> >
>>> >
>>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za(mailto:
marco@gallotta.co.za)> wrote:
>>> > > It's in /var which is persistent across reboots.
>>> > >
>>> > > --
>>> > > Marco Gallotta | Mountain View, California
>>> > > Software Engineer, Infrastructure | Loki Studios
>>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
twitter.com/marcog (http://twitter.com/marcog)
>>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
417-3313
>>> > >
>>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > >
>>> > >
>>> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>> > >
>>> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp
and you have
>>> > > > rebooted your machined then you will have problems.
>>> > > >
>>> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>> > > >
>>> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes
as we start
>>> > > > > gathering more data.
>>> > > > >
>>> > > > > I get the following error when running hbck -repair, and then
it stalls:
>>> > > > >
>>> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before
re-checking
>>> > > > > after fix...
>>> > > > > Version: 0.94.0
>>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo
from
>>> > > > > HDFS...
>>> > > > > Exception in thread "main"
java.util.concurrent.RejectedExecutionException
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>> > > > > at
>>> > > > >
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>> > > > > at
>>> > > > >
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>> > > > > at
>>> > > > >
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>>> > > > > at
>>> > > > >
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>>> > > > > at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>>> > > > > at
org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Marco Gallotta | Mountain View, California
>>> > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
417-3313
>>> > > > >
>>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > >
>>> > > > >
>>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>> > > > >
>>> > > > > > Is it a standalone installation or pseudo-distributed?
>>> > > > > > I faced a similar problem a few days back in a distributed
cluster and
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > used
>>> > > > > > hbck -repair option. You might give it a try.
>>> > > > > >
>>> > > > > > ~Anil
>>> > > > > >
>>> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>>> > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>> > > > > >
>>> > > > > > > Could you please share your /etc/hosts file??Meantime, do a
manual
>>> > > > > > > compaction and see if ti works.
>>> > > > > > >
>>> > > > > > > Regards,
>>> > > > > > > Mohammad Tariq
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>>> > > > > > > wrote:
>>> > > > > > > > It's not a distributed cluster. I'm not processing enough
data yet.
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > So
>>> > > > > > >
>>> > > > > > > the reference to localhost is correct.
>>> > > > > > > >
>>> > > > > > > > --
>>> > > > > > > > Marco Gallotta | Mountain View, California
>>> > > > > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
(650)
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > 417-3313
>>> > > > > > > >
>>> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>>> > > > > > > >
>>> > > > > > > > > Are you running a distributed cluster?
>>> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
>>> > > > > > > > >
>>> > > > > > > > > You are getting reference to localhost in hbck output:
>>> > > > > > > > > ERROR: Region { meta => null, hdfs =>
>>> > > > > > > > >
hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>>> > > > > > > > > deployed => } on HDFS, but not listed in META or
deployed on any
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > region
>>> > > > > > > > > server
>>> > > > > > > > >
>>> > > > > > > > > ~Anil
>>> > > > > > > > >
>>> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Here's the output from hbck -details:
>>> > > > > http://pastebin.com/ZxVZEctY
>>> > > > > > > > > >
>>> > > > > > > > > > Extract:
>>> > > > > > > > > >
>>> > > > > > > > > > 6 inconsistencies detected.
>>> > > > > > > > > > Status: INCONSISTENT
>>> > > > > > > > > >
>>> > > > > > > > > > 6 is the number of tables that appear in "list" but
cannot be
>>> > > > > > > operated on
>>> > > > > > > > > > (which btw, includes not being able to run
disable/drop on them -
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > both ops
>>> > > > > > > > > > say table not found). I also just noticed "foo" does
not occur
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > in a
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > table
>>> > > > > > > > > > list, although I did create it at one point but was
able to
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > clear it
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > from
>>> > > > > > > > > > .META. when it also was reporting table not found
when trying to
>>> > > > > > > > > > disable/drop it. All these come from when I ^C'ed
(i.e. killed)
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > table
>>> > > > > > > > > > creation when I was trying to get lzo compression
working and
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > table
>>> > > > > > > > > > creation was hanging.
>>> > > > > > > > > >
>>> > > > > > > > > > Is there any way to repair this? I see hbck has
repair options,
>>> > > > > but I
>>> > > > > > > want
>>> > > > > > > > > > to proceed with caution.
>>> > > > > > > > > >
>>> > > > > > > > > > --
>>> > > > > > > > > > Marco Gallotta | Mountain View, California
>>> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) |
+1 (650)
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 417-3313
>>> > > > > > > > > >
>>> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > > Hi Marco,
>>> > > > > > > > > > >
>>> > > > > > > > > > > Did anything disastrous happen to cluster?
>>> > > > > > > > > > > Can you try using hbck utility of HBase.
>>> > > > > > > > > > > Run: 'hbase hbck -help' to get all the available
options.
>>> > > > > > > > > > >
>>> > > > > > > > > > > ~Anil
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za
))>wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > > > Hi there
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > I have a few tables which show up in a "list" in
the shell,
>>> > > > > but
>>> > > > > > > produce
>>> > > > > > > > > > > > "table not found" when performing any operation
on them.
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > There is
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > no
>>> > > > > > > > > > > > reference of them in the .META. table. It seems
to be
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > resulting in
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > some of
>>> > > > > > > > > > > > the hbase services being killed every so often.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Here are some logs from master (foo is one of the
tables not
>>> > > > > > > found):
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>>> > > > > > > > > > > > Master server abort: loaded coprocessors are: []
>>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>>> > > > > > > > > > > > Unexpected state :
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>> > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > ..
>>> > > > > > > > > > > > Cannot transit it to OFFLINE.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > There are also a number of the following types of
error logs:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
>>> > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager:
Failed
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > assignment in:
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > due to
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>>> > > > > > > > > > > > Received:OPEN for the
>>> > > > > > > > > > > >
region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > ,which we
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > are
>>> > > > > > > > > > > > already trying to OPEN.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Any ideas how to find and remove any references
to these
>>> > > > > > > non-existent
>>> > > > > > > > > > > > tables?
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > --
>>> > > > > > > > > > > > Marco Gallotta | Mountain View, California
>>> > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta)
|
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)
| +1
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > (650)
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 417-3313
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Sent with Sparrow (
http://www.sparrowmailapp.com/?sig)
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > --
>>> > > > > > > > > > > Thanks & Regards,
>>> > > > > > > > > > > Anil Gupta
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > Thanks & Regards,
>>> > > > > > > > > Anil Gupta
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Thanks & Regards,
>>> > > > > > Anil Gupta
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Thanks & Regards,
>>> > > > Anil Gupta
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>>
>>>
>>>
>>
>>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
point your web browser to http://localhost:9000. It'll show all the
details of your HDFS.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> How do you check that?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>
>> This is pretty strange. I mean everything seems to be in place, but we
>> are stuck. Please make a check once if your Hdfs is in safemode.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
>> > What about fs.default.name (http://fs.default.name)?????
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
>> > > It's in /var which is persistent across reboots.
>> > >
>> > > --
>> > > Marco Gallotta | Mountain View, California
>> > > Software Engineer, Infrastructure | Loki Studios
>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > >
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > >
>> > >
>> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>> > >
>> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>> > > > rebooted your machined then you will have problems.
>> > > >
>> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>> > > >
>> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> > > > > gathering more data.
>> > > > >
>> > > > > I get the following error when running hbck -repair, and then it stalls:
>> > > > >
>> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> > > > > after fix...
>> > > > > Version: 0.94.0
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > > > > HDFS...
>> > > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Marco Gallotta | Mountain View, California
>> > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > > > >
>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > >
>> > > > >
>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > > > >
>> > > > > > Is it a standalone installation or pseudo-distributed?
>> > > > > > I faced a similar problem a few days back in a distributed cluster and
>> > > > > >
>> > > > >
>> > > > >
>> > > > > used
>> > > > > > hbck -repair option. You might give it a try.
>> > > > > >
>> > > > > > ~Anil
>> > > > > >
>> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>> > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>> > > > > >
>> > > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > > > > compaction and see if ti works.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Mohammad Tariq
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>> > > > > > > wrote:
>> > > > > > > > It's not a distributed cluster. I'm not processing enough data yet.
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > So
>> > > > > > >
>> > > > > > > the reference to localhost is correct.
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > 417-3313
>> > > > > > > >
>> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > > > > >
>> > > > > > > > > Are you running a distributed cluster?
>> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > > > > >
>> > > > > > > > > You are getting reference to localhost in hbck output:
>> > > > > > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > region
>> > > > > > > > > server
>> > > > > > > > >
>> > > > > > > > > ~Anil
>> > > > > > > > >
>> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > > > >
>> > > > > > > > > > Here's the output from hbck -details:
>> > > > > http://pastebin.com/ZxVZEctY
>> > > > > > > > > >
>> > > > > > > > > > Extract:
>> > > > > > > > > >
>> > > > > > > > > > 6 inconsistencies detected.
>> > > > > > > > > > Status: INCONSISTENT
>> > > > > > > > > >
>> > > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > > > > > operated on
>> > > > > > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > both ops
>> > > > > > > > > > say table not found). I also just noticed "foo" does not occur
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > in a
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > table
>> > > > > > > > > > list, although I did create it at one point but was able to
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > clear it
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > from
>> > > > > > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > table
>> > > > > > > > > > creation when I was trying to get lzo compression working and
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > table
>> > > > > > > > > > creation was hanging.
>> > > > > > > > > >
>> > > > > > > > > > Is there any way to repair this? I see hbck has repair options,
>> > > > > but I
>> > > > > > > want
>> > > > > > > > > > to proceed with caution.
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 417-3313
>> > > > > > > > > >
>> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Marco,
>> > > > > > > > > > >
>> > > > > > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > > > > > >
>> > > > > > > > > > > ~Anil
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi there
>> > > > > > > > > > > >
>> > > > > > > > > > > > I have a few tables which show up in a "list" in the shell,
>> > > > > but
>> > > > > > > produce
>> > > > > > > > > > > > "table not found" when performing any operation on them.
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > There is
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > no
>> > > > > > > > > > > > reference of them in the .META. table. It seems to be
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > resulting in
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > some of
>> > > > > > > > > > > > the hbase services being killed every so often.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > > > > > found):
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > > > > Master server abort: loaded coprocessors are: []
>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > > > > Unexpected state :
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > ..
>> > > > > > > > > > > > Cannot transit it to OFFLINE.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > There are also a number of the following types of error logs:
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
>> > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > assignment in:
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > due to
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > > > > > > > > > > Received:OPEN for the
>> > > > > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > ,which we
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > are
>> > > > > > > > > > > > already trying to OPEN.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Any ideas how to find and remove any references to these
>> > > > > > > non-existent
>> > > > > > > > > > > > tables?
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > (650)
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 417-3313
>> > > > > > > > > > > >
>> > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > Thanks & Regards,
>> > > > > > > > > > > Anil Gupta
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Thanks & Regards,
>> > > > > > > > > Anil Gupta
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Thanks & Regards,
>> > > > > > Anil Gupta
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Thanks & Regards,
>> > > > Anil Gupta
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
How do you check that?
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> This is pretty strange. I mean everything seems to be in place, but we
> are stuck. Please make a check once if your Hdfs is in safemode.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
> > What about fs.default.name (http://fs.default.name)?????
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > > It's in /var which is persistent across reboots.
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > >
> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > > > rebooted your machined then you will have problems.
> > > >
> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > > >
> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > > > gathering more data.
> > > > >
> > > > > I get the following error when running hbck -repair, and then it stalls:
> > > > >
> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > > > after fix...
> > > > > Version: 0.94.0
> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > > > HDFS...
> > > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > >
> > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > I faced a similar problem a few days back in a distributed cluster and
> > > > > >
> > > > >
> > > > >
> > > > > used
> > > > > > hbck -repair option. You might give it a try.
> > > > > >
> > > > > > ~Anil
> > > > > >
> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > > >
> > > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > > > > compaction and see if ti works.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Mohammad Tariq
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > > > wrote:
> > > > > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > So
> > > > > > >
> > > > > > > the reference to localhost is correct.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > 417-3313
> > > > > > > >
> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > >
> > > > > > > >
> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > >
> > > > > > > > > Are you running a distributed cluster?
> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > >
> > > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > region
> > > > > > > > > server
> > > > > > > > >
> > > > > > > > > ~Anil
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > >
> > > > > > > > > > Here's the output from hbck -details:
> > > > > http://pastebin.com/ZxVZEctY
> > > > > > > > > >
> > > > > > > > > > Extract:
> > > > > > > > > >
> > > > > > > > > > 6 inconsistencies detected.
> > > > > > > > > > Status: INCONSISTENT
> > > > > > > > > >
> > > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > > > operated on
> > > > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > both ops
> > > > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > in a
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > table
> > > > > > > > > > list, although I did create it at one point but was able to
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > clear it
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > from
> > > > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > table
> > > > > > > > > > creation when I was trying to get lzo compression working and
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > table
> > > > > > > > > > creation was hanging.
> > > > > > > > > >
> > > > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > > > but I
> > > > > > > want
> > > > > > > > > > to proceed with caution.
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 417-3313
> > > > > > > > > >
> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Marco,
> > > > > > > > > > >
> > > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > > > >
> > > > > > > > > > > ~Anil
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi there
> > > > > > > > > > > >
> > > > > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > > > > but
> > > > > > > produce
> > > > > > > > > > > > "table not found" when performing any operation on them.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > There is
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > no
> > > > > > > > > > > > reference of them in the .META. table. It seems to be
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > resulting in
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > some of
> > > > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > > > >
> > > > > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > > > > found):
> > > > > > > > > > > >
> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > > Unexpected state :
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ..
> > > > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > There are also a number of the following types of error logs:
> > > > > > > > > > > >
> > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > assignment in:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > due to
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > > > Received:OPEN for the
> > > > > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ,which we
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > are
> > > > > > > > > > > > already trying to OPEN.
> > > > > > > > > > > >
> > > > > > > > > > > > Any ideas how to find and remove any references to these
> > > > > > > non-existent
> > > > > > > > > > > > tables?
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > (650)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 417-3313
> > > > > > > > > > > >
> > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > Anil Gupta
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Anil Gupta
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks & Regards,
> > > > > > Anil Gupta
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> > >
> >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
This is pretty strange. I mean everything seems to be in place, but we
are stuck. Please make a check once if your Hdfs is in safemode.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <do...@gmail.com> wrote:
> What about fs.default.name?????
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
>> It's in /var which is persistent across reboots.
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> marco@gallotta.co.za | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>
>>> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>>> rebooted your machined then you will have problems.
>>>
>>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>>
>>> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>>> > gathering more data.
>>> >
>>> > I get the following error when running hbck -repair, and then it stalls:
>>> >
>>> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>>> > after fix...
>>> > Version: 0.94.0
>>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>>> > HDFS...
>>> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>>> > at
>>> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>>> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>>> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>> >
>>> >
>>> >
>>> > --
>>> > Marco Gallotta | Mountain View, California
>>> > Software Engineer, Infrastructure | Loki Studios
>>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>>> >
>>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> >
>>> >
>>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>> >
>>> > > Is it a standalone installation or pseudo-distributed?
>>> > > I faced a similar problem a few days back in a distributed cluster and
>>> > >
>>> >
>>> > used
>>> > > hbck -repair option. You might give it a try.
>>> > >
>>> > > ~Anil
>>> > >
>>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>>> > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>> > >
>>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>>> > > > compaction and see if ti works.
>>> > > >
>>> > > > Regards,
>>> > > > Mohammad Tariq
>>> > > >
>>> > > >
>>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>>> > > > wrote:
>>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > So
>>> > > >
>>> > > > the reference to localhost is correct.
>>> > > > >
>>> > > > > --
>>> > > > > Marco Gallotta | Mountain View, California
>>> > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>>> > > >
>>> > >
>>> >
>>> > 417-3313
>>> > > > >
>>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > >
>>> > > > >
>>> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>>> > > > >
>>> > > > > > Are you running a distributed cluster?
>>> > > > > > If yes, do you have localhost in /etc/hosts file?
>>> > > > > >
>>> > > > > > You are getting reference to localhost in hbck output:
>>> > > > > > ERROR: Region { meta => null, hdfs =>
>>> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>>> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> > region
>>> > > > > > server
>>> > > > > >
>>> > > > > > ~Anil
>>> > > > > >
>>> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>>> > > > > >
>>> > > > > > > Here's the output from hbck -details:
>>> > http://pastebin.com/ZxVZEctY
>>> > > > > > >
>>> > > > > > > Extract:
>>> > > > > > >
>>> > > > > > > 6 inconsistencies detected.
>>> > > > > > > Status: INCONSISTENT
>>> > > > > > >
>>> > > > > > > 6 is the number of tables that appear in "list" but cannot be
>>> > > > operated on
>>> > > > > > > (which btw, includes not being able to run disable/drop on them -
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > both ops
>>> > > > > > > say table not found). I also just noticed "foo" does not occur
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > in a
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > table
>>> > > > > > > list, although I did create it at one point but was able to
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > clear it
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > from
>>> > > > > > > .META. when it also was reporting table not found when trying to
>>> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > table
>>> > > > > > > creation when I was trying to get lzo compression working and
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> > table
>>> > > > > > > creation was hanging.
>>> > > > > > >
>>> > > > > > > Is there any way to repair this? I see hbck has repair options,
>>> > but I
>>> > > > want
>>> > > > > > > to proceed with caution.
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > Marco Gallotta | Mountain View, California
>>> > > > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > 417-3313
>>> > > > > > >
>>> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>> > > > > > >
>>> > > > > > > > Hi Marco,
>>> > > > > > > >
>>> > > > > > > > Did anything disastrous happen to cluster?
>>> > > > > > > > Can you try using hbck utility of HBase.
>>> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>>> > > > > > > >
>>> > > > > > > > ~Anil
>>> > > > > > > >
>>> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>>> > > > > > > >
>>> > > > > > > > > Hi there
>>> > > > > > > > >
>>> > > > > > > > > I have a few tables which show up in a "list" in the shell,
>>> > but
>>> > > > produce
>>> > > > > > > > > "table not found" when performing any operation on them.
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > There is
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > no
>>> > > > > > > > > reference of them in the .META. table. It seems to be
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > resulting in
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > some of
>>> > > > > > > > > the hbase services being killed every so often.
>>> > > > > > > > >
>>> > > > > > > > > Here are some logs from master (foo is one of the tables not
>>> > > > found):
>>> > > > > > > > >
>>> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>>> > > > org.apache.hadoop.hbase.master.HMaster:
>>> > > > > > > > > Master server abort: loaded coprocessors are: []
>>> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > org.apache.hadoop.hbase.master.HMaster:
>>> > > > > > > > > Unexpected state :
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>> > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > ..
>>> > > > > > > > > Cannot transit it to OFFLINE.
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > There are also a number of the following types of error logs:
>>> > > > > > > > >
>>> > > > > > > > > 2012-08-09 20:10:04,308 ERROR
>>> > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > assignment in:
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > due to
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>>> > > > > > > > > Received:OPEN for the
>>> > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > ,which we
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > are
>>> > > > > > > > > already trying to OPEN.
>>> > > > > > > > >
>>> > > > > > > > > Any ideas how to find and remove any references to these
>>> > > > non-existent
>>> > > > > > > > > tables?
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > Marco Gallotta | Mountain View, California
>>> > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>>> > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>>> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> > (650)
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > > 417-3313
>>> > > > > > > > >
>>> > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > --
>>> > > > > > > > Thanks & Regards,
>>> > > > > > > > Anil Gupta
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Thanks & Regards,
>>> > > > > > Anil Gupta
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Thanks & Regards,
>>> > > Anil Gupta
>>> > >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>>
>>>
>>
>>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:43 PM, Mohammad Tariq wrote:
> What about fs.default.name (http://fs.default.name)?????
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > It's in /var which is persistent across reboots.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> >
> > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > > rebooted your machined then you will have problems.
> > >
> > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > >
> > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > > gathering more data.
> > > >
> > > > I get the following error when running hbck -repair, and then it stalls:
> > > >
> > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > > after fix...
> > > > Version: 0.94.0
> > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > > HDFS...
> > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > > > at
> > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > >
> > > >
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > >
> > > > > Is it a standalone installation or pseudo-distributed?
> > > > > I faced a similar problem a few days back in a distributed cluster and
> > > > >
> > > >
> > > >
> > > > used
> > > > > hbck -repair option. You might give it a try.
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > > > >
> > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > > > compaction and see if ti works.
> > > > > >
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > > > wrote:
> > > > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > So
> > > > > >
> > > > > > the reference to localhost is correct.
> > > > > > >
> > > > > > > --
> > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > >
> > > > >
> > > >
> > > >
> > > > 417-3313
> > > > > > >
> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > >
> > > > > > >
> > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > >
> > > > > > > > Are you running a distributed cluster?
> > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > >
> > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > region
> > > > > > > > server
> > > > > > > >
> > > > > > > > ~Anil
> > > > > > > >
> > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > >
> > > > > > > > > Here's the output from hbck -details:
> > > > http://pastebin.com/ZxVZEctY
> > > > > > > > >
> > > > > > > > > Extract:
> > > > > > > > >
> > > > > > > > > 6 inconsistencies detected.
> > > > > > > > > Status: INCONSISTENT
> > > > > > > > >
> > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > > operated on
> > > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > both ops
> > > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > in a
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > table
> > > > > > > > > list, although I did create it at one point but was able to
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > clear it
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > from
> > > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > table
> > > > > > > > > creation when I was trying to get lzo compression working and
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > table
> > > > > > > > > creation was hanging.
> > > > > > > > >
> > > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > > but I
> > > > > > want
> > > > > > > > > to proceed with caution.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 417-3313
> > > > > > > > >
> > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > >
> > > > > > > > > > Hi Marco,
> > > > > > > > > >
> > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > > >
> > > > > > > > > > ~Anil
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi there
> > > > > > > > > > >
> > > > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > > > but
> > > > > > produce
> > > > > > > > > > > "table not found" when performing any operation on them.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > There is
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > no
> > > > > > > > > > > reference of them in the .META. table. It seems to be
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > resulting in
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > some of
> > > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > > >
> > > > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > > > found):
> > > > > > > > > > >
> > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > > > Unexpected state :
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ..
> > > > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > There are also a number of the following types of error logs:
> > > > > > > > > > >
> > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > assignment in:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > due to
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > > > Received:OPEN for the
> > > > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ,which we
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > are
> > > > > > > > > > > already trying to OPEN.
> > > > > > > > > > >
> > > > > > > > > > > Any ideas how to find and remove any references to these
> > > > > > non-existent
> > > > > > > > > > > tables?
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > (650)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 417-3313
> > > > > > > > > > >
> > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Thanks & Regards,
> > > > > > > > > > Anil Gupta
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks & Regards,
> > > > > > > > Anil Gupta
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Anil Gupta
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
What about fs.default.name?????
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> It's in /var which is persistent across reboots.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>
>> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>> rebooted your machined then you will have problems.
>>
>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>
>> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> > gathering more data.
>> >
>> > I get the following error when running hbck -repair, and then it stalls:
>> >
>> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> > after fix...
>> > Version: 0.94.0
>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > HDFS...
>> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> > at
>> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> >
>> >
>> >
>> > --
>> > Marco Gallotta | Mountain View, California
>> > Software Engineer, Infrastructure | Loki Studios
>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> >
>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> >
>> >
>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> >
>> > > Is it a standalone installation or pseudo-distributed?
>> > > I faced a similar problem a few days back in a distributed cluster and
>> > >
>> >
>> > used
>> > > hbck -repair option. You might give it a try.
>> > >
>> > > ~Anil
>> > >
>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>> > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>> > >
>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > compaction and see if ti works.
>> > > >
>> > > > Regards,
>> > > > Mohammad Tariq
>> > > >
>> > > >
>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>> > > > wrote:
>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>> > > >
>> > > >
>> > >
>> >
>> > So
>> > > >
>> > > > the reference to localhost is correct.
>> > > > >
>> > > > > --
>> > > > > Marco Gallotta | Mountain View, California
>> > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > >
>> > >
>> >
>> > 417-3313
>> > > > >
>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > >
>> > > > >
>> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > >
>> > > > > > Are you running a distributed cluster?
>> > > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > >
>> > > > > > You are getting reference to localhost in hbck output:
>> > > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> > region
>> > > > > > server
>> > > > > >
>> > > > > > ~Anil
>> > > > > >
>> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > >
>> > > > > > > Here's the output from hbck -details:
>> > http://pastebin.com/ZxVZEctY
>> > > > > > >
>> > > > > > > Extract:
>> > > > > > >
>> > > > > > > 6 inconsistencies detected.
>> > > > > > > Status: INCONSISTENT
>> > > > > > >
>> > > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > > operated on
>> > > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > both ops
>> > > > > > > say table not found). I also just noticed "foo" does not occur
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > in a
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > table
>> > > > > > > list, although I did create it at one point but was able to
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > clear it
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > from
>> > > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > table
>> > > > > > > creation when I was trying to get lzo compression working and
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> > table
>> > > > > > > creation was hanging.
>> > > > > > >
>> > > > > > > Is there any way to repair this? I see hbck has repair options,
>> > but I
>> > > > want
>> > > > > > > to proceed with caution.
>> > > > > > >
>> > > > > > > --
>> > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > 417-3313
>> > > > > > >
>> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > >
>> > > > > > >
>> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > > >
>> > > > > > > > Hi Marco,
>> > > > > > > >
>> > > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > > >
>> > > > > > > > ~Anil
>> > > > > > > >
>> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > > >
>> > > > > > > > > Hi there
>> > > > > > > > >
>> > > > > > > > > I have a few tables which show up in a "list" in the shell,
>> > but
>> > > > produce
>> > > > > > > > > "table not found" when performing any operation on them.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > There is
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > no
>> > > > > > > > > reference of them in the .META. table. It seems to be
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > resulting in
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > some of
>> > > > > > > > > the hbase services being killed every so often.
>> > > > > > > > >
>> > > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > > found):
>> > > > > > > > >
>> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > Master server abort: loaded coprocessors are: []
>> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > Unexpected state :
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > ..
>> > > > > > > > > Cannot transit it to OFFLINE.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > There are also a number of the following types of error logs:
>> > > > > > > > >
>> > > > > > > > > 2012-08-09 20:10:04,308 ERROR
>> > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > assignment in:
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > due to
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > > > > > > > Received:OPEN for the
>> > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > ,which we
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > are
>> > > > > > > > > already trying to OPEN.
>> > > > > > > > >
>> > > > > > > > > Any ideas how to find and remove any references to these
>> > > > non-existent
>> > > > > > > > > tables?
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>> > (650)
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > 417-3313
>> > > > > > > > >
>> > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Thanks & Regards,
>> > > > > > > > Anil Gupta
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Thanks & Regards,
>> > > > > > Anil Gupta
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks & Regards,
>> > > Anil Gupta
>> > >
>> >
>> >
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>>
>
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
It's in /var which is persistent across reboots.
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> rebooted your machined then you will have problems.
>
> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>
> > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > gathering more data.
> >
> > I get the following error when running hbck -repair, and then it stalls:
> >
> > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > after fix...
> > Version: 0.94.0
> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > HDFS...
> > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > at
> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> > at
> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > at
> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> > at
> > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> >
> >
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> >
> > > Is it a standalone installation or pseudo-distributed?
> > > I faced a similar problem a few days back in a distributed cluster and
> > >
> >
> > used
> > > hbck -repair option. You might give it a try.
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
> > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
> > >
> > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > compaction and see if ti works.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
> > > > wrote:
> > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > >
> > > >
> > >
> >
> > So
> > > >
> > > > the reference to localhost is correct.
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > >
> > > >
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > >
> > >
> >
> > 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > >
> > > > > > Are you running a distributed cluster?
> > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > >
> > > > > > You are getting reference to localhost in hbck output:
> > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> > region
> > > > > > server
> > > > > >
> > > > > > ~Anil
> > > > > >
> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > >
> > > > > > > Here's the output from hbck -details:
> > http://pastebin.com/ZxVZEctY
> > > > > > >
> > > > > > > Extract:
> > > > > > >
> > > > > > > 6 inconsistencies detected.
> > > > > > > Status: INCONSISTENT
> > > > > > >
> > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > operated on
> > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > >
> > > > >
> > > >
> > > >
> > > > both ops
> > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > in a
> > > > > >
> > > > >
> > > >
> > > >
> > > > table
> > > > > > > list, although I did create it at one point but was able to
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > clear it
> > > > > >
> > > > >
> > > >
> > > >
> > > > from
> > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > table
> > > > > > > creation when I was trying to get lzo compression working and
> > > > > >
> > > > >
> > > >
> > >
> >
> > table
> > > > > > > creation was hanging.
> > > > > > >
> > > > > > > Is there any way to repair this? I see hbck has repair options,
> > but I
> > > > want
> > > > > > > to proceed with caution.
> > > > > > >
> > > > > > > --
> > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > >
> > > > >
> > > >
> > > >
> > > > 417-3313
> > > > > > >
> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > >
> > > > > > >
> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > >
> > > > > > > > Hi Marco,
> > > > > > > >
> > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > >
> > > > > > > > ~Anil
> > > > > > > >
> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > > >
> > > > > > > > > Hi there
> > > > > > > > >
> > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > but
> > > > produce
> > > > > > > > > "table not found" when performing any operation on them.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > There is
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > no
> > > > > > > > > reference of them in the .META. table. It seems to be
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > resulting in
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > some of
> > > > > > > > > the hbase services being killed every so often.
> > > > > > > > >
> > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > found):
> > > > > > > > >
> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > > Unexpected state :
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ..
> > > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There are also a number of the following types of error logs:
> > > > > > > > >
> > > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > assignment in:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > due to
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > > Received:OPEN for the
> > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > ,which we
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > are
> > > > > > > > > already trying to OPEN.
> > > > > > > > >
> > > > > > > > > Any ideas how to find and remove any references to these
> > > > non-existent
> > > > > > > > > tables?
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > (650)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > 417-3313
> > > > > > > > >
> > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks & Regards,
> > > > > > > > Anil Gupta
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks & Regards,
> > > > > > Anil Gupta
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
Re: Table listed in "list", but not in .META.
Posted by anil gupta <an...@gmail.com>.
Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
rebooted your machined then you will have problems.
On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> gathering more data.
>
> I get the following error when running hbck -repair, and then it stalls:
>
> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> after fix...
> Version: 0.94.0
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> HDFS...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>
> > Is it a standalone installation or pseudo-distributed?
> > I faced a similar problem a few days back in a distributed cluster and
> used
> > hbck -repair option. You might give it a try.
> >
> > ~Anil
> >
> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com(mailto:
> dontariq@gmail.com)> wrote:
> >
> > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > compaction and see if ti works.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za(mailto:
> marco@gallotta.co.za)>
> > > wrote:
> > > > It's not a distributed cluster. I'm not processing enough data yet.
> So
> > >
> > > the reference to localhost is correct.
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > >
> > > > > Are you running a distributed cluster?
> > > > > If yes, do you have localhost in /etc/hosts file?
> > > > >
> > > > > You are getting reference to localhost in hbck output:
> > > > > ERROR: Region { meta => null, hdfs =>
> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > deployed => } on HDFS, but not listed in META or deployed on any
> region
> > > > > server
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > >
> > > > > > Here's the output from hbck -details:
> http://pastebin.com/ZxVZEctY
> > > > > >
> > > > > > Extract:
> > > > > >
> > > > > > 6 inconsistencies detected.
> > > > > > Status: INCONSISTENT
> > > > > >
> > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > operated on
> > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > >
> > > >
> > >
> > > both ops
> > > > > > say table not found). I also just noticed "foo" does not occur
> in a
> > > > >
> > > >
> > >
> > > table
> > > > > > list, although I did create it at one point but was able to
> clear it
> > > > >
> > > >
> > >
> > > from
> > > > > > .META. when it also was reporting table not found when trying to
> > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> table
> > > > > > creation when I was trying to get lzo compression working and
> table
> > > > > > creation was hanging.
> > > > > >
> > > > > > Is there any way to repair this? I see hbck has repair options,
> but I
> > > want
> > > > > > to proceed with caution.
> > > > > >
> > > > > > --
> > > > > > Marco Gallotta | Mountain View, California
> > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > >
> > > > >
> > > >
> > >
> > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > >
> > > >
> > >
> > > 417-3313
> > > > > >
> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > >
> > > > > >
> > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > >
> > > > > > > Hi Marco,
> > > > > > >
> > > > > > > Did anything disastrous happen to cluster?
> > > > > > > Can you try using hbck utility of HBase.
> > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > >
> > > > > > > ~Anil
> > > > > > >
> > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > > >
> > > > > > > > Hi there
> > > > > > > >
> > > > > > > > I have a few tables which show up in a "list" in the shell,
> but
> > > produce
> > > > > > > > "table not found" when performing any operation on them.
> There is
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > no
> > > > > > > > reference of them in the .META. table. It seems to be
> resulting in
> > > > > > >
> > > > > >
> > > > > >
> > > > > > some of
> > > > > > > > the hbase services being killed every so often.
> > > > > > > >
> > > > > > > > Here are some logs from master (foo is one of the tables not
> > > found):
> > > > > > > >
> > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > > Unexpected state :
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > ..
> > > > > > > > Cannot transit it to OFFLINE.
> > > > > > > >
> > > > > > > >
> > > > > > > > There are also a number of the following types of error logs:
> > > > > > > >
> > > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > assignment in:
> > > > > > > >
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > due to
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > > Received:OPEN for the
> > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > ,which we
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > are
> > > > > > > > already trying to OPEN.
> > > > > > > >
> > > > > > > > Any ideas how to find and remove any references to these
> > > non-existent
> > > > > > > > tables?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
> (650)
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > 417-3313
> > > > > > > >
> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Thanks & Regards,
> > > > > > > Anil Gupta
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Anil Gupta
> > > > >
> > > >
> > >
> > >
> >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
> >
>
>
>
--
Thanks & Regards,
Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
It's a pseudo-distributed cluster, as I plan to add more nodes as we start gathering more data.
I get the following error when running hbck -repair, and then it stalls:
12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking after fix...
Version: 0.94.0
12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
Exception in thread "main" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> Is it a standalone installation or pseudo-distributed?
> I faced a similar problem a few days back in a distributed cluster and used
> hbck -repair option. You might give it a try.
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
>
> > Could you please share your /etc/hosts file??Meantime, do a manual
> > compaction and see if ti works.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
> > wrote:
> > > It's not a distributed cluster. I'm not processing enough data yet. So
> >
> > the reference to localhost is correct.
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > >
> > > > Are you running a distributed cluster?
> > > > If yes, do you have localhost in /etc/hosts file?
> > > >
> > > > You are getting reference to localhost in hbck output:
> > > > ERROR: Region { meta => null, hdfs =>
> > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > > server
> > > >
> > > > ~Anil
> > > >
> > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > >
> > > > > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> > > > >
> > > > > Extract:
> > > > >
> > > > > 6 inconsistencies detected.
> > > > > Status: INCONSISTENT
> > > > >
> > > > > 6 is the number of tables that appear in "list" but cannot be
> > operated on
> > > > > (which btw, includes not being able to run disable/drop on them -
> > > >
> > >
> >
> > both ops
> > > > > say table not found). I also just noticed "foo" does not occur in a
> > > >
> > >
> >
> > table
> > > > > list, although I did create it at one point but was able to clear it
> > > >
> > >
> >
> > from
> > > > > .META. when it also was reporting table not found when trying to
> > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > > creation when I was trying to get lzo compression working and table
> > > > > creation was hanging.
> > > > >
> > > > > Is there any way to repair this? I see hbck has repair options, but I
> > want
> > > > > to proceed with caution.
> > > > >
> > > > > --
> > > > > Marco Gallotta | Mountain View, California
> > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > >
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > >
> > >
> >
> > 417-3313
> > > > >
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > >
> > > > > > Hi Marco,
> > > > > >
> > > > > > Did anything disastrous happen to cluster?
> > > > > > Can you try using hbck utility of HBase.
> > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > >
> > > > > > ~Anil
> > > > > >
> > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > > >
> > > > > > > Hi there
> > > > > > >
> > > > > > > I have a few tables which show up in a "list" in the shell, but
> > produce
> > > > > > > "table not found" when performing any operation on them. There is
> > > > > >
> > > > >
> > > >
> > >
> >
> > no
> > > > > > > reference of them in the .META. table. It seems to be resulting in
> > > > > >
> > > > >
> > > > >
> > > > > some of
> > > > > > > the hbase services being killed every so often.
> > > > > > >
> > > > > > > Here are some logs from master (foo is one of the tables not
> > found):
> > > > > > >
> > > > > > > 2012-08-09 20:40:44,301 FATAL
> > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > Master server abort: loaded coprocessors are: []
> > > > > > > 2012-08-09 20:40:44,301 FATAL
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.master.HMaster:
> > > > > > > Unexpected state :
> > > > > >
> > > > >
> > > >
> > >
> >
> > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > >
> > > > >
> > > >
> > >
> >
> > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > ..
> > > > > > > Cannot transit it to OFFLINE.
> > > > > > >
> > > > > > >
> > > > > > > There are also a number of the following types of error logs:
> > > > > > >
> > > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> > assignment in:
> > > > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > >
> > > > >
> > > >
> > >
> >
> > due to
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > > Received:OPEN for the
> > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ,which we
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > are
> > > > > > > already trying to OPEN.
> > > > > > >
> > > > > > > Any ideas how to find and remove any references to these
> > non-existent
> > > > > > > tables?
> > > > > > >
> > > > > > > --
> > > > > > > Marco Gallotta | Mountain View, California
> > > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> > 417-3313
> > > > > > >
> > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks & Regards,
> > > > > > Anil Gupta
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> >
> >
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
Re: Table listed in "list", but not in .META.
Posted by anil gupta <an...@gmail.com>.
Is it a standalone installation or pseudo-distributed?
I faced a similar problem a few days back in a distributed cluster and used
hbck -repair option. You might give it a try.
~Anil
On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za>
> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So
> the reference to localhost is correct.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta | twitter.com/marcog
> > marco@gallotta.co.za | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> >
> >> Are you running a distributed cluster?
> >> If yes, do you have localhost in /etc/hosts file?
> >>
> >> You are getting reference to localhost in hbck output:
> >> ERROR: Region { meta => null, hdfs =>
> >> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> >> deployed => } on HDFS, but not listed in META or deployed on any region
> >> server
> >>
> >> ~Anil
> >>
> >> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za(mailto:
> marco@gallotta.co.za)>wrote:
> >>
> >> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> >> >
> >> > Extract:
> >> >
> >> > 6 inconsistencies detected.
> >> > Status: INCONSISTENT
> >> >
> >> > 6 is the number of tables that appear in "list" but cannot be
> operated on
> >> > (which btw, includes not being able to run disable/drop on them -
> both ops
> >> > say table not found). I also just noticed "foo" does not occur in a
> table
> >> > list, although I did create it at one point but was able to clear it
> from
> >> > .META. when it also was reporting table not found when trying to
> >> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> >> > creation when I was trying to get lzo compression working and table
> >> > creation was hanging.
> >> >
> >> > Is there any way to repair this? I see hbck has repair options, but I
> want
> >> > to proceed with caution.
> >> >
> >> > --
> >> > Marco Gallotta | Mountain View, California
> >> > Software Engineer, Infrastructure | Loki Studios
> >> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> >> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> 417-3313
> >> >
> >> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >> >
> >> >
> >> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> >> >
> >> > > Hi Marco,
> >> > >
> >> > > Did anything disastrous happen to cluster?
> >> > > Can you try using hbck utility of HBase.
> >> > > Run: 'hbase hbck -help' to get all the available options.
> >> > >
> >> > > ~Anil
> >> > >
> >> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> >> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> >> > >
> >> > > > Hi there
> >> > > >
> >> > > > I have a few tables which show up in a "list" in the shell, but
> produce
> >> > > > "table not found" when performing any operation on them. There is
> no
> >> > > > reference of them in the .META. table. It seems to be resulting in
> >> > > >
> >> > >
> >> > >
> >> >
> >> > some of
> >> > > > the hbase services being killed every so often.
> >> > > >
> >> > > > Here are some logs from master (foo is one of the tables not
> found):
> >> > > >
> >> > > > 2012-08-09 20:40:44,301 FATAL
> org.apache.hadoop.hbase.master.HMaster:
> >> > > > Master server abort: loaded coprocessors are: []
> >> > > > 2012-08-09 20:40:44,301 FATAL
> org.apache.hadoop.hbase.master.HMaster:
> >> > > > Unexpected state :
> foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> >> > > > state=PENDING_OPEN, ts=1344570044277,
> >> > > >
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> >> > > >
> >> > >
> >> >
> >> > ..
> >> > > > Cannot transit it to OFFLINE.
> >> > > >
> >> > > >
> >> > > > There are also a number of the following types of error logs:
> >> > > >
> >> > > > 2012-08-09 20:10:04,308 ERROR
> >> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
> assignment in:
> >> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> due to
> >> > > >
> >> > >
> >> >
> >> >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> >> > > > Received:OPEN for the
> >> > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> ,which we
> >> > > >
> >> > >
> >> >
> >> > are
> >> > > > already trying to OPEN.
> >> > > >
> >> > > > Any ideas how to find and remove any references to these
> non-existent
> >> > > > tables?
> >> > > >
> >> > > > --
> >> > > > Marco Gallotta | Mountain View, California
> >> > > > Software Engineer, Infrastructure | Loki Studios
> >> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> >> > > >
> >> > >
> >> >
> >> > twitter.com/marcog (http://twitter.com/marcog)
> >> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
> 417-3313
> >> > > >
> >> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >> > >
> >> > >
> >> > > --
> >> > > Thanks & Regards,
> >> > > Anil Gupta
> >> > >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Anil Gupta
> >>
> >>
> >
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
Nothing perculiar in /etc/hosts:
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Ran a major compaction with
echo 'list' | ./hbase shell | tail -n52 | head -n50 | while read t; do echo "major_compact '$t'"; done | ./hbase shell
And still get the 6 inconsistencies.
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 3:39 PM, Mohammad Tariq wrote:
> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
>
> Regards,
> Mohammad Tariq
>
>
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> >
> > > Are you running a distributed cluster?
> > > If yes, do you have localhost in /etc/hosts file?
> > >
> > > You are getting reference to localhost in hbck output:
> > > ERROR: Region { meta => null, hdfs =>
> > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > server
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
> > >
> > > > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> > > >
> > > > Extract:
> > > >
> > > > 6 inconsistencies detected.
> > > > Status: INCONSISTENT
> > > >
> > > > 6 is the number of tables that appear in "list" but cannot be operated on
> > > > (which btw, includes not being able to run disable/drop on them - both ops
> > > > say table not found). I also just noticed "foo" does not occur in a table
> > > > list, although I did create it at one point but was able to clear it from
> > > > .META. when it also was reporting table not found when trying to
> > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > creation when I was trying to get lzo compression working and table
> > > > creation was hanging.
> > > >
> > > > Is there any way to repair this? I see hbck has repair options, but I want
> > > > to proceed with caution.
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > >
> > > > > Hi Marco,
> > > > >
> > > > > Did anything disastrous happen to cluster?
> > > > > Can you try using hbck utility of HBase.
> > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > >
> > > > > ~Anil
> > > > >
> > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > > > >
> > > > > > Hi there
> > > > > >
> > > > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > > > "table not found" when performing any operation on them. There is no
> > > > > > reference of them in the .META. table. It seems to be resulting in
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > some of
> > > > > > the hbase services being killed every so often.
> > > > > >
> > > > > > Here are some logs from master (foo is one of the tables not found):
> > > > > >
> > > > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > > > Master server abort: loaded coprocessors are: []
> > > > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > > > state=PENDING_OPEN, ts=1344570044277,
> > > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > > > >
> > > > >
> > > >
> > > >
> > > > ..
> > > > > > Cannot transit it to OFFLINE.
> > > > > >
> > > > > >
> > > > > > There are also a number of the following types of error logs:
> > > > > >
> > > > > > 2012-08-09 20:10:04,308 ERROR
> > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > > > > >
> > > > >
> > > >
> > > >
> > > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > > > Received:OPEN for the
> > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> > > > > >
> > > > >
> > > >
> > > >
> > > > are
> > > > > > already trying to OPEN.
> > > > > >
> > > > > > Any ideas how to find and remove any references to these non-existent
> > > > > > tables?
> > > > > >
> > > > > > --
> > > > > > Marco Gallotta | Mountain View, California
> > > > > > Software Engineer, Infrastructure | Loki Studios
> > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > > > >
> > > > >
> > > >
> > > >
> > > > twitter.com/marcog (http://twitter.com/marcog)
> > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > > > >
> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Anil Gupta
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
>
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
Could you please share your /etc/hosts file??Meantime, do a manual
compaction and see if ti works.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>
>> Are you running a distributed cluster?
>> If yes, do you have localhost in /etc/hosts file?
>>
>> You are getting reference to localhost in hbck output:
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> deployed => } on HDFS, but not listed in META or deployed on any region
>> server
>>
>> ~Anil
>>
>> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>
>> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>> >
>> > Extract:
>> >
>> > 6 inconsistencies detected.
>> > Status: INCONSISTENT
>> >
>> > 6 is the number of tables that appear in "list" but cannot be operated on
>> > (which btw, includes not being able to run disable/drop on them - both ops
>> > say table not found). I also just noticed "foo" does not occur in a table
>> > list, although I did create it at one point but was able to clear it from
>> > .META. when it also was reporting table not found when trying to
>> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> > creation when I was trying to get lzo compression working and table
>> > creation was hanging.
>> >
>> > Is there any way to repair this? I see hbck has repair options, but I want
>> > to proceed with caution.
>> >
>> > --
>> > Marco Gallotta | Mountain View, California
>> > Software Engineer, Infrastructure | Loki Studios
>> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> >
>> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> >
>> >
>> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> >
>> > > Hi Marco,
>> > >
>> > > Did anything disastrous happen to cluster?
>> > > Can you try using hbck utility of HBase.
>> > > Run: 'hbase hbck -help' to get all the available options.
>> > >
>> > > ~Anil
>> > >
>> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > >
>> > > > Hi there
>> > > >
>> > > > I have a few tables which show up in a "list" in the shell, but produce
>> > > > "table not found" when performing any operation on them. There is no
>> > > > reference of them in the .META. table. It seems to be resulting in
>> > > >
>> > >
>> > >
>> >
>> > some of
>> > > > the hbase services being killed every so often.
>> > > >
>> > > > Here are some logs from master (foo is one of the tables not found):
>> > > >
>> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > > Master server abort: loaded coprocessors are: []
>> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > >
>> > >
>> >
>> > ..
>> > > > Cannot transit it to OFFLINE.
>> > > >
>> > > >
>> > > > There are also a number of the following types of error logs:
>> > > >
>> > > > 2012-08-09 20:10:04,308 ERROR
>> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
>> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
>> > > >
>> > >
>> >
>> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > > Received:OPEN for the
>> > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
>> > > >
>> > >
>> >
>> > are
>> > > > already trying to OPEN.
>> > > >
>> > > > Any ideas how to find and remove any references to these non-existent
>> > > > tables?
>> > > >
>> > > > --
>> > > > Marco Gallotta | Mountain View, California
>> > > > Software Engineer, Infrastructure | Loki Studios
>> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > >
>> > >
>> >
>> > twitter.com/marcog (http://twitter.com/marcog)
>> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > > >
>> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > >
>> > >
>> > > --
>> > > Thanks & Regards,
>> > > Anil Gupta
>> > >
>> >
>> >
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>>
>
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
>
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>
> > Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
> >
> > Extract:
> >
> > 6 inconsistencies detected.
> > Status: INCONSISTENT
> >
> > 6 is the number of tables that appear in "list" but cannot be operated on
> > (which btw, includes not being able to run disable/drop on them - both ops
> > say table not found). I also just noticed "foo" does not occur in a table
> > list, although I did create it at one point but was able to clear it from
> > .META. when it also was reporting table not found when trying to
> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > creation when I was trying to get lzo compression working and table
> > creation was hanging.
> >
> > Is there any way to repair this? I see hbck has repair options, but I want
> > to proceed with caution.
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> >
> > > Hi Marco,
> > >
> > > Did anything disastrous happen to cluster?
> > > Can you try using hbck utility of HBase.
> > > Run: 'hbase hbck -help' to get all the available options.
> > >
> > > ~Anil
> > >
> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
> > >
> > > > Hi there
> > > >
> > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > "table not found" when performing any operation on them. There is no
> > > > reference of them in the .META. table. It seems to be resulting in
> > > >
> > >
> > >
> >
> > some of
> > > > the hbase services being killed every so often.
> > > >
> > > > Here are some logs from master (foo is one of the tables not found):
> > > >
> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Master server abort: loaded coprocessors are: []
> > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > > state=PENDING_OPEN, ts=1344570044277,
> > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> > > >
> > >
> >
> > ..
> > > > Cannot transit it to OFFLINE.
> > > >
> > > >
> > > > There are also a number of the following types of error logs:
> > > >
> > > > 2012-08-09 20:10:04,308 ERROR
> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > > >
> > >
> >
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > > Received:OPEN for the
> > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> > > >
> > >
> >
> > are
> > > > already trying to OPEN.
> > > >
> > > > Any ideas how to find and remove any references to these non-existent
> > > > tables?
> > > >
> > > > --
> > > > Marco Gallotta | Mountain View, California
> > > > Software Engineer, Infrastructure | Loki Studios
> > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> > > >
> > >
> >
> > twitter.com/marcog (http://twitter.com/marcog)
> > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > > >
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
Re: Table listed in "list", but not in .META.
Posted by Mohammad Tariq <do...@gmail.com>.
@Anil : Good Point.
@Marco : First make sure that all the AMIs running region servers are
reachable and there is no problem in DNS resolution.(As I see you are
using AWS).
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 4:00 AM, anil gupta <an...@gmail.com> wrote:
> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
>
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
>
>> Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>>
>> Extract:
>>
>> 6 inconsistencies detected.
>> Status: INCONSISTENT
>>
>> 6 is the number of tables that appear in "list" but cannot be operated on
>> (which btw, includes not being able to run disable/drop on them - both ops
>> say table not found). I also just noticed "foo" does not occur in a table
>> list, although I did create it at one point but was able to clear it from
>> .META. when it also was reporting table not found when trying to
>> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> creation when I was trying to get lzo compression working and table
>> creation was hanging.
>>
>> Is there any way to repair this? I see hbck has repair options, but I want
>> to proceed with caution.
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta | twitter.com/marcog
>> marco@gallotta.co.za | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>
>> > Hi Marco,
>> >
>> > Did anything disastrous happen to cluster?
>> > Can you try using hbck utility of HBase.
>> > Run: 'hbase hbck -help' to get all the available options.
>> >
>> > ~Anil
>> >
>> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za(mailto:
>> marco@gallotta.co.za)>wrote:
>> >
>> > > Hi there
>> > >
>> > > I have a few tables which show up in a "list" in the shell, but produce
>> > > "table not found" when performing any operation on them. There is no
>> > > reference of them in the .META. table. It seems to be resulting in
>> some of
>> > > the hbase services being killed every so often.
>> > >
>> > > Here are some logs from master (foo is one of the tables not found):
>> > >
>> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > Master server abort: loaded coprocessors are: []
>> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
>> > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > state=PENDING_OPEN, ts=1344570044277,
>> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> ..
>> > > Cannot transit it to OFFLINE.
>> > >
>> > >
>> > > There are also a number of the following types of error logs:
>> > >
>> > > 2012-08-09 20:10:04,308 ERROR
>> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
>> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
>> > >
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > Received:OPEN for the
>> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
>> are
>> > > already trying to OPEN.
>> > >
>> > > Any ideas how to find and remove any references to these non-existent
>> > > tables?
>> > >
>> > > --
>> > > Marco Gallotta | Mountain View, California
>> > > Software Engineer, Infrastructure | Loki Studios
>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> twitter.com/marcog (http://twitter.com/marcog)
>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > >
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Anil Gupta
>> >
>> >
>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by anil gupta <an...@gmail.com>.
Are you running a distributed cluster?
If yes, do you have localhost in /etc/hosts file?
You are getting reference to localhost in hbck output:
ERROR: Region { meta => null, hdfs =>
hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
deployed => } on HDFS, but not listed in META or deployed on any region
server
~Anil
On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
>
> Extract:
>
> 6 inconsistencies detected.
> Status: INCONSISTENT
>
> 6 is the number of tables that appear in "list" but cannot be operated on
> (which btw, includes not being able to run disable/drop on them - both ops
> say table not found). I also just noticed "foo" does not occur in a table
> list, although I did create it at one point but was able to clear it from
> .META. when it also was reporting table not found when trying to
> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> creation when I was trying to get lzo compression working and table
> creation was hanging.
>
> Is there any way to repair this? I see hbck has repair options, but I want
> to proceed with caution.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>
> > Hi Marco,
> >
> > Did anything disastrous happen to cluster?
> > Can you try using hbck utility of HBase.
> > Run: 'hbase hbck -help' to get all the available options.
> >
> > ~Anil
> >
> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za(mailto:
> marco@gallotta.co.za)>wrote:
> >
> > > Hi there
> > >
> > > I have a few tables which show up in a "list" in the shell, but produce
> > > "table not found" when performing any operation on them. There is no
> > > reference of them in the .META. table. It seems to be resulting in
> some of
> > > the hbase services being killed every so often.
> > >
> > > Here are some logs from master (foo is one of the tables not found):
> > >
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Master server abort: loaded coprocessors are: []
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > > state=PENDING_OPEN, ts=1344570044277,
> > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
> ..
> > > Cannot transit it to OFFLINE.
> > >
> > >
> > > There are also a number of the following types of error logs:
> > >
> > > 2012-08-09 20:10:04,308 ERROR
> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > Received:OPEN for the
> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> are
> > > already trying to OPEN.
> > >
> > > Any ideas how to find and remove any references to these non-existent
> > > tables?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
> >
>
>
>
--
Thanks & Regards,
Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
Here's the output from hbck -details: http://pastebin.com/ZxVZEctY
Extract:
6 inconsistencies detected.
Status: INCONSISTENT
6 is the number of tables that appear in "list" but cannot be operated on (which btw, includes not being able to run disable/drop on them - both ops say table not found). I also just noticed "foo" does not occur in a table list, although I did create it at one point but was able to clear it from .META. when it also was reporting table not found when trying to disable/drop it. All these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging.
Is there any way to repair this? I see hbck has repair options, but I want to proceed with caution.
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> Hi Marco,
>
> Did anything disastrous happen to cluster?
> Can you try using hbck utility of HBase.
> Run: 'hbase hbck -help' to get all the available options.
>
> ~Anil
>
> On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>
> > Hi there
> >
> > I have a few tables which show up in a "list" in the shell, but produce
> > "table not found" when performing any operation on them. There is no
> > reference of them in the .META. table. It seems to be resulting in some of
> > the hbase services being killed every so often.
> >
> > Here are some logs from master (foo is one of the tables not found):
> >
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Master server abort: loaded coprocessors are: []
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> > state=PENDING_OPEN, ts=1344570044277,
> > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> > Cannot transit it to OFFLINE.
> >
> >
> > There are also a number of the following types of error logs:
> >
> > 2012-08-09 20:10:04,308 ERROR
> > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > Received:OPEN for the
> > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> > already trying to OPEN.
> >
> > Any ideas how to find and remove any references to these non-existent
> > tables?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
Re: Table listed in "list", but not in .META.
Posted by anil gupta <an...@gmail.com>.
Hi Marco,
Did anything disastrous happen to cluster?
Can you try using hbck utility of HBase.
Run: 'hbase hbck -help' to get all the available options.
~Anil
On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce
> "table not found" when performing any operation on them. There is no
> reference of them in the .META. table. It seems to be resulting in some of
> the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent
> tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
--
Thanks & Regards,
Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by anil gupta <an...@gmail.com>.
Good to hear that... :) I also had a very bad crash of HBase a few days
ago... at that time had to run hbck repair to fix some errors and to fix
others i had to remove files from /hbase dir. It's a fairly complex system
so sometimes it really hard to nail the problem.
~Anil
On Fri, Aug 10, 2012 at 8:21 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> w00t! I looked in "/hbase/" on the hdfs and found directories referencing
> the stale tables and deleting them worked. It now says 0 inconsistencies.
> Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone
> for the help! Hopefully this stops the sporadic crashes I was seeing.
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote:
>
> > I never did that, but I mentioned above that all these come from when I
> ^C'ed (i.e. killed) table creation when I was trying to get lzo compression
> working and table creation was hanging. Where would I check for references
> to these tables in ZK?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog(
> http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
> >
> > > Did you blow away your cluster once (in the sense of the deleting all
> data and starting from scratch)?This could be left-over data in ZK.
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Marco Gallotta <marco@gallotta.co.za (mailto:
> marco@gallotta.co.za)>
> > > To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> > > Cc:
> > > Sent: Friday, August 10, 2012 2:22 PM
> > > Subject: Table listed in "list", but not in .META.
> > >
> > > Hi there
> > >
> > > I have a few tables which show up in a "list" in the shell, but
> produce "table not found" when performing any operation on them. There is
> no reference of them in the .META. table. It seems to be resulting in some
> of the hbase services being killed every so often.
> > >
> > > Here are some logs from master (foo is one of the tables not found):
> > >
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
> > >
> > >
> > > There are also a number of the following types of error logs:
> > >
> > > 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
> > >
> > > Any ideas how to find and remove any references to these non-existent
> tables?
> > >
> > > --
> > > Marco Gallotta | Mountain View, California
> > > Software Engineer, Infrastructure | Loki Studios
> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
> twitter.com/marcog (http://twitter.com/marcog)
> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> > >
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
>
>
--
Thanks & Regards,
Anil Gupta
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
w00t! I looked in "/hbase/" on the hdfs and found directories referencing the stale tables and deleting them worked. It now says 0 inconsistencies. Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone for the help! Hopefully this stops the sporadic crashes I was seeing.
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote:
> I never did that, but I mentioned above that all these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging. Where would I check for references to these tables in ZK?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
>
> > Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.
> >
> >
> >
> > ----- Original Message -----
> > From: Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
> > To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> > Cc:
> > Sent: Friday, August 10, 2012 2:22 PM
> > Subject: Table listed in "list", but not in .META.
> >
> > Hi there
> >
> > I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
> >
> > Here are some logs from master (foo is one of the tables not found):
> >
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
> >
> >
> > There are also a number of the following types of error logs:
> >
> > 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
> >
> > Any ideas how to find and remove any references to these non-existent tables?
> >
> > --
> > Marco Gallotta | Mountain View, California
> > Software Engineer, Infrastructure | Loki Studios
> > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
> >
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
Re: Table listed in "list", but not in .META.
Posted by Marco Gallotta <ma...@gallotta.co.za>.
I never did that, but I mentioned above that all these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging. Where would I check for references to these tables in ZK?
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote:
> Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.
>
>
>
> ----- Original Message -----
> From: Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
> To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> Cc:
> Sent: Friday, August 10, 2012 2:22 PM
> Subject: Table listed in "list", but not in .META.
>
> Hi there
>
> I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
>
> Here are some logs from master (foo is one of the tables not found):
>
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
>
>
> There are also a number of the following types of error logs:
>
> 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
>
> Any ideas how to find and remove any references to these non-existent tables?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
Re: Table listed in "list", but not in .META.
Posted by lars hofhansl <lh...@yahoo.com>.
Did you blow away your cluster once (in the sense of the deleting all data and starting from scratch)?This could be left-over data in ZK.
----- Original Message -----
From: Marco Gallotta <ma...@gallotta.co.za>
To: user@hbase.apache.org
Cc:
Sent: Friday, August 10, 2012 2:22 PM
Subject: Table listed in "list", but not in .META.
Hi there
I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
Here are some logs from master (foo is one of the tables not found):
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:
2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
Any ideas how to find and remove any references to these non-existent tables?
--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
marco@gallotta.co.za | +1 (650) 417-3313
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)