You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Saad Mufti <sa...@gmail.com> on 2018/03/21 03:00:39 UTC
Should Taking A Snapshot Work Even If Balancer Is Moving A Few
Regions Around?
Hi,
We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in S3,
they take much longer than when using local disk. We have a cron script to
take regular snapshots as backup, and they fail quite often on our largest
table which takes close to an hour to complete the snapshot.
The only thing I have noticed in the errors usually is a message about the
region moving or closing.
So my question is whether taking a snapshot is supposed to work even with
regions being moved around. In our case it is usually only a couple here
and there.
Thanks.
----
Saad
Re: Should Taking A Snapshot Work Even If Balancer Is Moving A Few
Regions Around?
Posted by Saad Mufti <sa...@gmail.com>.
Thanks.
----
Saad
On Wed, Mar 21, 2018 at 2:33 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >>So my question is whether taking a snapshot is supposed to work even with
> >>regions being moved around. In our case it is usually only a couple here
> >>and there.
>
> No, if region was moved, split or merged during snapshot operation -
> snapshot will fail.
> This is why taking snapshots on a large table is a 50/50 game.
>
> Disabling balancer,region merging and split before snapshot should help.
> This works in 2.0
>
> Not sure if merge/split switch is available in 1.4
>
> -Vlad
>
> On Tue, Mar 20, 2018 at 8:00 PM, Saad Mufti <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in
> S3,
> > they take much longer than when using local disk. We have a cron script
> to
> > take regular snapshots as backup, and they fail quite often on our
> largest
> > table which takes close to an hour to complete the snapshot.
> >
> > The only thing I have noticed in the errors usually is a message about
> the
> > region moving or closing.
> >
> > So my question is whether taking a snapshot is supposed to work even with
> > regions being moved around. In our case it is usually only a couple here
> > and there.
> >
> > Thanks.
> >
> > ----
> > Saad
> >
>
Re: Should Taking A Snapshot Work Even If Balancer Is Moving A Few
Regions Around?
Posted by Saad Mufti <sa...@gmail.com>.
Thanks.
----
Saad
On Wed, Mar 21, 2018 at 3:04 PM, Ted Yu <yu...@gmail.com> wrote:
> Looking at
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java in
> branch-1.4 :
>
> boolean[] setSplitOrMergeEnabled(final boolean enabled, final boolean
> synchronous,
> final MasterSwitchType... switchTypes)
> throws IOException;
>
> boolean isSplitOrMergeEnabled(final MasterSwitchType switchType) throws
> IOException;
>
> Please also see the following script:
>
> hbase-shell/src/main/ruby/shell/commands/splitormerge_switch.rb
>
> FYI
>
> On Wed, Mar 21, 2018 at 11:33 AM, Vladimir Rodionov <
> vladrodionov@gmail.com>
> wrote:
>
> > >>So my question is whether taking a snapshot is supposed to work even
> with
> > >>regions being moved around. In our case it is usually only a couple
> here
> > >>and there.
> >
> > No, if region was moved, split or merged during snapshot operation -
> > snapshot will fail.
> > This is why taking snapshots on a large table is a 50/50 game.
> >
> > Disabling balancer,region merging and split before snapshot should help.
> > This works in 2.0
> >
> > Not sure if merge/split switch is available in 1.4
> >
> > -Vlad
> >
> > On Tue, Mar 20, 2018 at 8:00 PM, Saad Mufti <sa...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in
> > S3,
> > > they take much longer than when using local disk. We have a cron script
> > to
> > > take regular snapshots as backup, and they fail quite often on our
> > largest
> > > table which takes close to an hour to complete the snapshot.
> > >
> > > The only thing I have noticed in the errors usually is a message about
> > the
> > > region moving or closing.
> > >
> > > So my question is whether taking a snapshot is supposed to work even
> with
> > > regions being moved around. In our case it is usually only a couple
> here
> > > and there.
> > >
> > > Thanks.
> > >
> > > ----
> > > Saad
> > >
> >
>
Re: Should Taking A Snapshot Work Even If Balancer Is Moving A Few
Regions Around?
Posted by Ted Yu <yu...@gmail.com>.
Looking at
hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java in
branch-1.4 :
boolean[] setSplitOrMergeEnabled(final boolean enabled, final boolean
synchronous,
final MasterSwitchType... switchTypes)
throws IOException;
boolean isSplitOrMergeEnabled(final MasterSwitchType switchType) throws
IOException;
Please also see the following script:
hbase-shell/src/main/ruby/shell/commands/splitormerge_switch.rb
FYI
On Wed, Mar 21, 2018 at 11:33 AM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >>So my question is whether taking a snapshot is supposed to work even with
> >>regions being moved around. In our case it is usually only a couple here
> >>and there.
>
> No, if region was moved, split or merged during snapshot operation -
> snapshot will fail.
> This is why taking snapshots on a large table is a 50/50 game.
>
> Disabling balancer,region merging and split before snapshot should help.
> This works in 2.0
>
> Not sure if merge/split switch is available in 1.4
>
> -Vlad
>
> On Tue, Mar 20, 2018 at 8:00 PM, Saad Mufti <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in
> S3,
> > they take much longer than when using local disk. We have a cron script
> to
> > take regular snapshots as backup, and they fail quite often on our
> largest
> > table which takes close to an hour to complete the snapshot.
> >
> > The only thing I have noticed in the errors usually is a message about
> the
> > region moving or closing.
> >
> > So my question is whether taking a snapshot is supposed to work even with
> > regions being moved around. In our case it is usually only a couple here
> > and there.
> >
> > Thanks.
> >
> > ----
> > Saad
> >
>
Re: Should Taking A Snapshot Work Even If Balancer Is Moving A Few
Regions Around?
Posted by Vladimir Rodionov <vl...@gmail.com>.
>>So my question is whether taking a snapshot is supposed to work even with
>>regions being moved around. In our case it is usually only a couple here
>>and there.
No, if region was moved, split or merged during snapshot operation -
snapshot will fail.
This is why taking snapshots on a large table is a 50/50 game.
Disabling balancer,region merging and split before snapshot should help.
This works in 2.0
Not sure if merge/split switch is available in 1.4
-Vlad
On Tue, Mar 20, 2018 at 8:00 PM, Saad Mufti <sa...@gmail.com> wrote:
> Hi,
>
> We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in S3,
> they take much longer than when using local disk. We have a cron script to
> take regular snapshots as backup, and they fail quite often on our largest
> table which takes close to an hour to complete the snapshot.
>
> The only thing I have noticed in the errors usually is a message about the
> region moving or closing.
>
> So my question is whether taking a snapshot is supposed to work even with
> regions being moved around. In our case it is usually only a couple here
> and there.
>
> Thanks.
>
> ----
> Saad
>