You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andrey Timerbaev <at...@gmx.net> on 2010/09/26 11:21:51 UTC
How to make HBase redundant?
Dear All,
In my Hadoop 0.20.1/HBase 0.20.3 cluster I’ve got 2 region servers. On each
region server, Hadoop DataNode and HBase RegionServer processes are running.
Hadoop is set up redundantly (it writes each data to both DataNodes, i.e. writes
data with replication factor 2). But as per information from the HBase Web UI,
HBase creates on the region servers unique regions. E.g. tables A, B are created
on region server 1, and tables C, D are created on the region server 2. Next, I
noticed, when region server 1 is not available, the tables A and B are not
available as well (cannot be read/written to).
Could you kindly suggest, how to make HBase redundant, so that all tables are
available even if one of region servers fails/dies?
Have I missed a Hadoop/HBase setting to do so?
Thank you,
Andrey Timerbaev
Re: How to make HBase redundant?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
|On Wed, Sep 29, 2010 at 3:49 AM, Matthew LeMieux <md...@mlogiciels.com> wrote:
> The problem I referred to was also being addressed in a separate thread, thanks to the contributors to the mailing list and mostly to J-D and Stack.
>
> I have recently upgraded to the 0.89.20100924 version and after more than 24 hours am very happy with the results. I think I must have missed the previous announcement of this particular version being available. (which list should I join to get such announcements?).
The announcements for the release candidates are made on dev@, once
the vote passes it will be announced on user@
>
> In the end, I would like to thank them for their timely, relevant and ingenious efforts in response to the posts on the mailing list.
Trying our best :)
J-D
Re: How to make HBase redundant?
Posted by Matthew LeMieux <md...@mlogiciels.com>.
The problem I referred to was also being addressed in a separate thread, thanks to the contributors to the mailing list and mostly to J-D and Stack.
I have recently upgraded to the 0.89.20100924 version and after more than 24 hours am very happy with the results. I think I must have missed the previous announcement of this particular version being available. (which list should I join to get such announcements?).
In the end, I would like to thank them for their timely, relevant and ingenious efforts in response to the posts on the mailing list.
-Matthew
O n Sep 27, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>> That's funny. My understanding was, region servers were redundant inherently. If
>> they are "semiredundant", there should be a root cause like some wrong settings
>> or a bug.
>>
>> Could someone from HBase experts comment on this?
>
> 0.89 is a developer release, it should be treated as such (eg do
> expect bugs) and this is the version used by Matthew. A newer release
> candidate was posted here:
> http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
> and this is the version we're using in production (and on a few other
> clusters) at StumbleUpon. We can kill -9 region servers as much as we
> want, and the cluster does recover. Previously there was an issue with
> empty log files that's now fixed, also 0.89.2010830 introduced a
> changed that's incompatible with the old way of recovering edits from
> a failed region server, so if leftovers from a previous split are
> present this can prevent the master from splitting logs at all (which
> is another issue Matthew got in a separate thread).
>
> J-D
Re: How to make HBase redundant?
Posted by Andrey Timerbaev <at...@gmx.net>.
Jean-Daniel Cryans <jd...@...> writes:
> We can kill -9 region servers as much as we
> want, and the cluster does recover.
>
> J-D
Just FYI: I've just tested the redundancy of region servers on a Hadoop
0.20.2/HBase 0.20.6, with 2 region servers. The results are: it works fine, only
if the region to be killed doesn't contain ROOT and META regions. In the
opposite case, i.e. if the killed region contained the ROOT, cluster is not
operable after killing it
(org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0). Neither after
the killed region server has been brought up again. Both reads and writes failed.
As result, per my tests, the region servers are not redundant in 0.20.6.
Andrey
Re: How to make HBase redundant?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
>
> Does the 0.20.6 provide the redundancy of region servers as well as the
> 0.89.20100924?
It generally does, but since HDFS 0.20 didn't support fsSync then
testing region server recovery is pretty hard (since you never know
how much data you lost). But for 0.90, which the 0.89 releases are
snapshots of, the goal is no data loss and 100% recovery. Bugs that
are found in that regard always have high priority.
J-D
Re: How to make HBase redundant?
Posted by Andrey Timerbaev <at...@gmx.net>.
Jean-Daniel Cryans <jd...@...> writes:
> 0.89 is a developer release, it should be treated as such (eg do
> expect bugs) and this is the version used by Matthew. A newer release
> candidate was posted here:
> http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
> and this is the version we're using in production (and on a few other
> clusters) at StumbleUpon. We can kill -9 region servers as much as we
> want, and the cluster does recover.
Hello Jean-Daniel,
Does the 0.20.6 provide the redundancy of region servers as well as the
0.89.20100924?
Thank you,
Andrey
Re: How to make HBase redundant?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> That's funny. My understanding was, region servers were redundant inherently. If
> they are "semiredundant", there should be a root cause like some wrong settings
> or a bug.
>
> Could someone from HBase experts comment on this?
0.89 is a developer release, it should be treated as such (eg do
expect bugs) and this is the version used by Matthew. A newer release
candidate was posted here:
http://people.apache.org/~jdcryans/hbase-0.89.20100924-candidate-1/
and this is the version we're using in production (and on a few other
clusters) at StumbleUpon. We can kill -9 region servers as much as we
want, and the cluster does recover. Previously there was an issue with
empty log files that's now fixed, also 0.89.2010830 introduced a
changed that's incompatible with the old way of recovering edits from
a failed region server, so if leftovers from a previous split are
present this can prevent the master from splitting logs at all (which
is another issue Matthew got in a separate thread).
J-D
Re: How to make HBase redundant?
Posted by Andrey Timerbaev <at...@gmx.net>.
Matthew LeMieux <md...@...> writes:
>
> It is the default behavior. However, in my installation it has never
succeeded. So, the regions never
> actually get re-assigned. The result is a cluster in a funky state that
needs to be completely brought down
> and then started up again in order to recover.
>
> I'm running on a version of HBase that is several weeks old, but I've never
>seen the cluster recover from a region server going on previous versions
>either.
That's funny. My understanding was, region servers were redundant inherently. If
they are "semiredundant", there should be a root cause like some wrong settings
or a bug.
Could someone from HBase experts comment on this?
Re: How to make HBase redundant?
Posted by Matthew LeMieux <md...@mlogiciels.com>.
It is the default behavior. However, in my installation it has never succeeded. So, the regions never actually get re-assigned. The result is a cluster in a funky state that needs to be completely brought down and then started up again in order to recover.
I'm running on a version of HBase that is several weeks old, but I've never seen the cluster recover from a region server going on previous versions either. I'm hoping this will be fixed soon.
You should see the Master making the attempt at the same time the region server goes down in the logs. In my case, it complains about this: "java.io.IOException: cannot get log writer" caused by this "java.io.FileNotFoundException: Parent path is not a directory".
FYI,
Matthew
On Sep 26, 2010, at 11:23 PM, Andrey Timerbaev wrote:
> Stanislaw Kogut <sk...@...> writes:
>
>> In case when one of regionservers fail, HBase Master will reassign all
>> regions, so, any regions served by failed regionserver will be
>> assigned to operating regionservers. It is possible because of
>> replication on datanodes, regionserver heartbeats and
>> Write-Ahead-Logs. HBase master can be made redundant too.
>
> Hello Stanislaw,
>
> This is a good news. Is a particular setting needed to do HBase behave in the
> redundant manner, or is it an out-of-box behavior? (In my installation I didn't
> notice namely, that HBase reassigns the regions.)
>
> Thank you,
> Andrey
>
>
>
>
Re: How to make HBase redundant?
Posted by Andrey Timerbaev <at...@gmx.net>.
Stanislaw Kogut <sk...@...> writes:
> In case when one of regionservers fail, HBase Master will reassign all
> regions, so, any regions served by failed regionserver will be
> assigned to operating regionservers. It is possible because of
> replication on datanodes, regionserver heartbeats and
> Write-Ahead-Logs. HBase master can be made redundant too.
Hello Stanislaw,
This is a good news. Is a particular setting needed to do HBase behave in the
redundant manner, or is it an out-of-box behavior? (In my installation I didn't
notice namely, that HBase reassigns the regions.)
Thank you,
Andrey
Re: How to make HBase redundant?
Posted by Stanislaw Kogut <sk...@sistyma.net>.
On Sun, Sep 26, 2010 at 12:21 PM, Andrey Timerbaev <at...@gmx.net> wrote:
> Dear All,
>
> In my Hadoop 0.20.1/HBase 0.20.3 cluster I’ve got 2 region servers. On each
> region server, Hadoop DataNode and HBase RegionServer processes are running.
> Hadoop is set up redundantly (it writes each data to both DataNodes, i.e. writes
> data with replication factor 2). But as per information from the HBase Web UI,
> HBase creates on the region servers unique regions. E.g. tables A, B are created
> on region server 1, and tables C, D are created on the region server 2. Next, I
> noticed, when region server 1 is not available, the tables A and B are not
> available as well (cannot be read/written to).
>
> Could you kindly suggest, how to make HBase redundant, so that all tables are
> available even if one of region servers fails/dies?
> Have I missed a Hadoop/HBase setting to do so?
>
> Thank you,
> Andrey Timerbaev
>
>
>
In case when one of regionservers fail, HBase Master will reassign all
regions, so, any regions served by failed regionserver will be
assigned to operating regionservers. It is possible because of
replication on datanodes, regionserver heartbeats and
Write-Ahead-Logs. HBase master can be made redundant too.
--
Regards,
Stanislaw Kogut
Sistyma LLC