HBase on EMR disk space issue

[Oussama Jilal <ji...@gmail.com>]

Hello, I need some help with my issue,

We have been running HBase on Amazon EMR (v 0.92) for quite some months, 
today we had a serious problem, Hbase was throwing some "connection 
refused" errors in out client applications, after some investigation, we 
found out that one of the two EC2 instances running hbase (the 
slave/regionserver one) was disk full (800+ G), we can connect to both 
of the instances, but when we try to run a command from the HBase Shell 
(list, scan, disable, truncate ... anything) we only get errors like :

13/07/22 14:06:29 INFO httpclient.HttpMethodDirector: I/O exception 
(java.net.ConnectException) caught when processing request: Connection 
refused
13/07/22 14:06:29 INFO httpclient.HttpMethodDirector: Retrying request

We want to truncate some tables and gz compress some column families but 
we can't do any operation.

Executing the command "jps" on the master returns :

6966 Jps
1968 JobTracker
2177 HQuorumPeer
1814 NameNode
4623 HMaster

but on the region server it only return :

5986 Jps

Rebooting didn't help ...

Any one can help ? We are not very experienced with HBase, especially 
administration.

Thank you.

Re: HBase on EMR disk space issue

[Marcos Luis Ortiz Valmaseda <ma...@gmail.com>]

Regards, Oussana.
First advice: Update your HBase installation to a more recent version
like 0.94.9 for example:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12324431

The problem with EMR is the version for HBase is very old, so, it
would be nice instead EC2 for HBase with a recent version.
See the official docummentation here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hbase-launch.html

See these tips too:
http://aws-musings.com/7-tips-for-running-hbase-in-ec2/

HBase use-cases in the last HBaseCon 2013.
http://www.orzota.com/hbase-use-cases/
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/35001


My favorite is the HBase at Pinterest talk:
http://www.hbasecon.com/sessions/apache-hbase-operations-at-pinterest/
http://www.slideshare.net/cloudera/operations-session-1

2013/7/22, Oussama Jilal <ji...@gmail.com>:
> Hello, I need some help with my issue,
>
> We have been running HBase on Amazon EMR (v 0.92) for quite some months,
> today we had a serious problem, Hbase was throwing some "connection
> refused" errors in out client applications, after some investigation, we
> found out that one of the two EC2 instances running hbase (the
> slave/regionserver one) was disk full (800+ G), we can connect to both
> of the instances, but when we try to run a command from the HBase Shell
> (list, scan, disable, truncate ... anything) we only get errors like :
>
> 13/07/22 14:06:29 INFO httpclient.HttpMethodDirector: I/O exception
> (java.net.ConnectException) caught when processing request: Connection
> refused
> 13/07/22 14:06:29 INFO httpclient.HttpMethodDirector: Retrying request
>
> We want to truncate some tables and gz compress some column families but
> we can't do any operation.
>
> Executing the command "jps" on the master returns :
>
> 6966 Jps
> 1968 JobTracker
> 2177 HQuorumPeer
> 1814 NameNode
> 4623 HMaster
>
> but on the region server it only return :
>
> 5986 Jps
>
> Rebooting didn't help ...
>
> Any one can help ? We are not very experienced with HBase, especially
> administration.
>
> Thank you.
>


-- 
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz