You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Lex Toumbourou <le...@scrunch.com> on 2016/05/05 03:53:34 UTC
Export HBase snapshot to S3 creates empty root directory (prefix)
Hi all,
I'm having a couple of problems with exporting HBase snapshots to S3. I am
running HBase version 1.2.0.
I have a table called "domain"
And I have created a snapshot for it:
hbase(main):003:0> snapshot 'domain', 'domain-aws-test'
0 row(s) in 0.3310 seconds
---
I am attempting to export it to S3 using the following command:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
"domain-aws-test" -copy-to s3://my-hbase-snapshots/domain-aws-test
Now, when I view the snapshot metadata in the S3 bucket, there's nothing
there:
> aws s3 ls my-hbase-snapshots/domain-aws-snapshots
But there is data under:
> aws s3 ls my-hbase-snapshots/\/domain-aws-test/
PRE .hbase-snapshot/
2016-05-05 13:38:12 1 .hbase-snapshot
It seems what's happening is, HBase is creating a directory/prefix with no
name and placing the snapshot data under there.
That wouldn't be a problem, except that when I try to import the snapshot
on my destination cluster, it seems unable to deal with the empty directory:
sudo -u hbase hbase snapshot export -D
hbase.rootdir=s3://my-hbase-snapshots/\/domain-aws-test -snapshot
my-aws-test -copy-to hdfs://hbaseClusterDNSName:8020/user/hbase -mappers 2
Caused by: java.io.FileNotFoundException: No such file or directory
's3://my-hbase-snapshots/domain-aws-test/.hbase-snapshot/domain-aws-test'
---
Any one come across this before?
Lex
Re: Export HBase snapshot to S3 creates empty root directory (prefix)
Posted by Lex Toumbourou <le...@scrunch.com>.
Thank you sooo much guys! I used s3n and that's fixed it. I will try s3a
too. Thanks for much for that thread, Ted.
On 5 May 2016 at 14:07, Ted Yu <yu...@gmail.com> wrote:
> Lex:
> Please also see this thread about s3n versus s3a:
>
> http://search-hadoop.com/m/uOzYtE1Fy22eEWfe1&subj=Re+S3+Hadoop+FileSystems
>
> On Wed, May 4, 2016 at 9:01 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > never seen that problem before, but a couple of suggestions you can try.
> >
> > Instead of the old s3 driver, you can use s3n or s3a if you have it
> > available (those are the ones I tested)
> > and instead of using hbase.root dir use -copy-from
> >
> > ExportSnapshot -snapshot SNAPSHOT_NAME -copy-to s3a://BUCKET/NAMESPACE
> > ExportSnapshot -snapshot SNAPSHOT_NAME -copy-from s3a://BUCKET/NAMESPACE
> > -copy-to hdfs://HOST/hbase
> >
> > you can take a look at some in-progress doc about s3 and snapshot here:
> > https://issues.apache.org/jira/browse/HBASE-15646
> >
> > Matteo
> >
> >
> > On Wed, May 4, 2016 at 8:53 PM, Lex Toumbourou <le...@scrunch.com> wrote:
> >
> > > Hi all,
> > >
> > > I'm having a couple of problems with exporting HBase snapshots to S3. I
> > am
> > > running HBase version 1.2.0.
> > >
> > > I have a table called "domain"
> > >
> > > And I have created a snapshot for it:
> > >
> > > hbase(main):003:0> snapshot 'domain', 'domain-aws-test'
> > > 0 row(s) in 0.3310 seconds
> > >
> > > ---
> > >
> > > I am attempting to export it to S3 using the following command:
> > >
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > "domain-aws-test" -copy-to s3://my-hbase-snapshots/domain-aws-test
> > >
> > > Now, when I view the snapshot metadata in the S3 bucket, there's
> nothing
> > > there:
> > >
> > > > aws s3 ls my-hbase-snapshots/domain-aws-snapshots
> > >
> > > But there is data under:
> > >
> > > > aws s3 ls my-hbase-snapshots/\/domain-aws-test/
> > > PRE .hbase-snapshot/
> > > 2016-05-05 13:38:12 1 .hbase-snapshot
> > >
> > > It seems what's happening is, HBase is creating a directory/prefix with
> > no
> > > name and placing the snapshot data under there.
> > >
> > > That wouldn't be a problem, except that when I try to import the
> snapshot
> > > on my destination cluster, it seems unable to deal with the empty
> > > directory:
> > >
> > > sudo -u hbase hbase snapshot export -D
> > > hbase.rootdir=s3://my-hbase-snapshots/\/domain-aws-test -snapshot
> > > my-aws-test -copy-to hdfs://hbaseClusterDNSName:8020/user/hbase
> -mappers
> > 2
> > >
> > > Caused by: java.io.FileNotFoundException: No such file or directory
> > >
> 's3://my-hbase-snapshots/domain-aws-test/.hbase-snapshot/domain-aws-test'
> > >
> > > ---
> > >
> > > Any one come across this before?
> > >
> > > Lex
> > >
> >
>
--
Lex ToumbourouLead engineer at scrunch.com <http://scrunch.com/>
Re: Export HBase snapshot to S3 creates empty root directory (prefix)
Posted by Ted Yu <yu...@gmail.com>.
Lex:
Please also see this thread about s3n versus s3a:
http://search-hadoop.com/m/uOzYtE1Fy22eEWfe1&subj=Re+S3+Hadoop+FileSystems
On Wed, May 4, 2016 at 9:01 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> never seen that problem before, but a couple of suggestions you can try.
>
> Instead of the old s3 driver, you can use s3n or s3a if you have it
> available (those are the ones I tested)
> and instead of using hbase.root dir use -copy-from
>
> ExportSnapshot -snapshot SNAPSHOT_NAME -copy-to s3a://BUCKET/NAMESPACE
> ExportSnapshot -snapshot SNAPSHOT_NAME -copy-from s3a://BUCKET/NAMESPACE
> -copy-to hdfs://HOST/hbase
>
> you can take a look at some in-progress doc about s3 and snapshot here:
> https://issues.apache.org/jira/browse/HBASE-15646
>
> Matteo
>
>
> On Wed, May 4, 2016 at 8:53 PM, Lex Toumbourou <le...@scrunch.com> wrote:
>
> > Hi all,
> >
> > I'm having a couple of problems with exporting HBase snapshots to S3. I
> am
> > running HBase version 1.2.0.
> >
> > I have a table called "domain"
> >
> > And I have created a snapshot for it:
> >
> > hbase(main):003:0> snapshot 'domain', 'domain-aws-test'
> > 0 row(s) in 0.3310 seconds
> >
> > ---
> >
> > I am attempting to export it to S3 using the following command:
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > "domain-aws-test" -copy-to s3://my-hbase-snapshots/domain-aws-test
> >
> > Now, when I view the snapshot metadata in the S3 bucket, there's nothing
> > there:
> >
> > > aws s3 ls my-hbase-snapshots/domain-aws-snapshots
> >
> > But there is data under:
> >
> > > aws s3 ls my-hbase-snapshots/\/domain-aws-test/
> > PRE .hbase-snapshot/
> > 2016-05-05 13:38:12 1 .hbase-snapshot
> >
> > It seems what's happening is, HBase is creating a directory/prefix with
> no
> > name and placing the snapshot data under there.
> >
> > That wouldn't be a problem, except that when I try to import the snapshot
> > on my destination cluster, it seems unable to deal with the empty
> > directory:
> >
> > sudo -u hbase hbase snapshot export -D
> > hbase.rootdir=s3://my-hbase-snapshots/\/domain-aws-test -snapshot
> > my-aws-test -copy-to hdfs://hbaseClusterDNSName:8020/user/hbase -mappers
> 2
> >
> > Caused by: java.io.FileNotFoundException: No such file or directory
> > 's3://my-hbase-snapshots/domain-aws-test/.hbase-snapshot/domain-aws-test'
> >
> > ---
> >
> > Any one come across this before?
> >
> > Lex
> >
>
Re: Export HBase snapshot to S3 creates empty root directory (prefix)
Posted by Matteo Bertozzi <th...@gmail.com>.
never seen that problem before, but a couple of suggestions you can try.
Instead of the old s3 driver, you can use s3n or s3a if you have it
available (those are the ones I tested)
and instead of using hbase.root dir use -copy-from
ExportSnapshot -snapshot SNAPSHOT_NAME -copy-to s3a://BUCKET/NAMESPACE
ExportSnapshot -snapshot SNAPSHOT_NAME -copy-from s3a://BUCKET/NAMESPACE
-copy-to hdfs://HOST/hbase
you can take a look at some in-progress doc about s3 and snapshot here:
https://issues.apache.org/jira/browse/HBASE-15646
Matteo
On Wed, May 4, 2016 at 8:53 PM, Lex Toumbourou <le...@scrunch.com> wrote:
> Hi all,
>
> I'm having a couple of problems with exporting HBase snapshots to S3. I am
> running HBase version 1.2.0.
>
> I have a table called "domain"
>
> And I have created a snapshot for it:
>
> hbase(main):003:0> snapshot 'domain', 'domain-aws-test'
> 0 row(s) in 0.3310 seconds
>
> ---
>
> I am attempting to export it to S3 using the following command:
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> "domain-aws-test" -copy-to s3://my-hbase-snapshots/domain-aws-test
>
> Now, when I view the snapshot metadata in the S3 bucket, there's nothing
> there:
>
> > aws s3 ls my-hbase-snapshots/domain-aws-snapshots
>
> But there is data under:
>
> > aws s3 ls my-hbase-snapshots/\/domain-aws-test/
> PRE .hbase-snapshot/
> 2016-05-05 13:38:12 1 .hbase-snapshot
>
> It seems what's happening is, HBase is creating a directory/prefix with no
> name and placing the snapshot data under there.
>
> That wouldn't be a problem, except that when I try to import the snapshot
> on my destination cluster, it seems unable to deal with the empty
> directory:
>
> sudo -u hbase hbase snapshot export -D
> hbase.rootdir=s3://my-hbase-snapshots/\/domain-aws-test -snapshot
> my-aws-test -copy-to hdfs://hbaseClusterDNSName:8020/user/hbase -mappers 2
>
> Caused by: java.io.FileNotFoundException: No such file or directory
> 's3://my-hbase-snapshots/domain-aws-test/.hbase-snapshot/domain-aws-test'
>
> ---
>
> Any one come across this before?
>
> Lex
>