You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by oc tsdb <oc...@gmail.com> on 2013/12/02 18:19:34 UTC
HBase ExportSnapshot
Hi,
We have cluster with 4 data nodes and HBase version is 0.94.10.
We have created snapshot for all hbase tables and trying to export snapshot
in two ways.
option 1.Export snapshot into same cluster hdfs
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16;
Here we are getting full data ( .archive + .hbase-snapshot) exported to
hdfs:/hbase_backup
option 2.Export snapshot to local filesystem
command :
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers 16;
But with option 2 we only getting .hbase-snapshot exported to local dir
(/tmp/hbase_backup) but .archive files are not exported.It is expected
behavior or something wrong in option 2.
Thanks
OC
Re: HBase ExportSnapshot
Posted by oc tsdb <oc...@gmail.com>.
I think that error is because of missing .archive files?
We only see this issue when we export snapshot to local file system
(file:///tmp/hbase_backup)
but not when we export snapshot to hdfs.
When we export snapshot to hdfs we are getting all the data.Its only not
working when we export snapshot data to local file system directly?
On Tue, Dec 3, 2013 at 5:39 PM, Ted Yu <yu...@gmail.com> wrote:
> Please check what happened to the HFileLink mentioned in the exception
> below - this would be the cause for snapshot export failure:
>
> Exception in thread "main" java.io.FileNotFoundException: Unable to open
> link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ]
> at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
>
>
> On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb <oc...@gmail.com> wrote:
>
> > here is snapshot export logs.
> >
> > mastre log:
> > ===========
> > 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> >
> > snapshot export console log:
> > =============================
> >
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> >
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > 13/12/02 21:54:24 INFO util.FSVisitor: No families under region
> > directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
> > 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
> > 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
> > be copied. There may be no data in the table.
> > 13/12/02 21:54:25 INFO util.FSVisitor: No families under region
> > directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
> > 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
> > 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
> > be copied. There may be no data in the table.
> > Exception in thread "main" java.io.FileNotFoundException: Unable to open
> > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ,
> > hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ,
> > hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ]
> > at
> > org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> > at
> >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> > at
> >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> >
> > Basically while exporting to local i dont see .archive directory.why?
> >
> > Please comment on this -
> > It seems we can't export complete snapshot data directly to local file
> > system using 'ExportSnapshot' command.
> > If we want to copy to outside of cluster first we need to export it to
> hdfs
> > and then use hadoop get command to copy to local file system.
> > Is this correct?
> >
> > Thanks
> > -OC
> >
> >
> > On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > The log you pasted was for taking snapshot.
> > > Do you have log from ExportSnapshot ?
> > >
> > > bq. What is the difference between below two commands?
> > >
> > > This is the code that determines output FileSystem:
> > >
> > > FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
> > >
> > > For 'file:///tmp/hbase_backup' argument, outputFs would be an instance
> of
> > > org.apache.hadoop.fs.LocalFileSystem.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc...@gmail.com> wrote:
> > >
> > > > We see same logs for both options
> > > >
> > > > 013-12-02 09:47:41,311 INFO
> > > > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> > > FLUSH
> > > > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > > FileSystem doesn't support getDefaultReplication
> > > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > > FileSystem doesn't support getDefaultBlockSize
> > > > 2013-12-02 09:47:41,337 INFO
> > org.apache.hadoop.hbase.procedure.Procedure:
> > > > Starting procedure 'tsdb_snap_backup'
> > > > 2013-12-02 09:47:41,724 INFO
> > org.apache.hadoop.hbase.procedure.Procedure:
> > > > Procedure 'tsdb_snap_backup' execution completed
> > > > 2013-12-02 09:47:41,724 INFO
> > > > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all
> znodes
> > > for
> > > > procedure tsdb_snap_backupincluding nodes
> > /hbase/online-snapshot/acquired
> > > > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > > > 2013-12-02 09:47:41,730 INFO
> > > > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler:
> > Done
> > > > waiting - snapshot for tsdb_snap_backup finished!
> > > >
> > > > It seems we can't export complete snapshot data directly to local
> file
> > > > system using 'ExportSnapshot' command.
> > > > If we want to copy to outside of cluster first we need to export it
> to
> > > hdfs
> > > > and then use hadoop get command to copy to local file system.
> > > > Is this correct?
> > > >
> > > > What is the difference between below two commands?
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers
> > 16;
> > > >
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16;
> > > >
> > > > Thanks
> > > > -OC
> > > >
> > > >
> > > >
> > > > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Can you pastebin master log during operation #2 ?
> > > > >
> > > > > There have been at least two fixes since 0.94.10, listed below.
> > > > > It would be nice if you can verify this behavior using 0.94.14
> > > > >
> > > > > Cheers
> > > > >
> > > > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013)
> > | 1
> > > > > line
> > > > >
> > > > > HBASE-8760 possible loss of data in snapshot taken after region
> split
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013)
> > | 1
> > > > > line
> > > > >
> > > > > HBASE-9060 ExportSnapshot job fails if target path contains
> > percentage
> > > > > character (Jerry He)
> > > > >
> > > > >
> > > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > > > >
> > > > > > We have created snapshot for all hbase tables and trying to
> export
> > > > > snapshot
> > > > > > in two ways.
> > > > > >
> > > > > > option 1.Export snapshot into same cluster hdfs
> > > > > >
> > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers
> > 16;
> > > > > >
> > > > > > Here we are getting full data ( .archive + .hbase-snapshot)
> > exported
> > > to
> > > > > > hdfs:/hbase_backup
> > > > > >
> > > > > > option 2.Export snapshot to local filesystem
> > > > > > command :
> > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup*
> > > -mappers
> > > > > 16;
> > > > > >
> > > > > > But with option 2 we only getting .hbase-snapshot exported to
> local
> > > dir
> > > > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> > > expected
> > > > > > behavior or something wrong in option 2.
> > > > > >
> > > > > > Thanks
> > > > > > OC
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HBase ExportSnapshot
Posted by Ted Yu <yu...@gmail.com>.
Please check what happened to the HFileLink mentioned in the exception
below - this would be the cause for snapshot export failure:
Exception in thread "main" java.io.FileNotFoundException: Unable to open
link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
]
at
org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb <oc...@gmail.com> wrote:
> here is snapshot export logs.
>
> mastre log:
> ===========
> 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
>
> snapshot export console log:
> =============================
>
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
>
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> 13/12/02 21:54:24 INFO util.FSVisitor: No families under region
> directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
> 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
> 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
> be copied. There may be no data in the table.
> 13/12/02 21:54:25 INFO util.FSVisitor: No families under region
> directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
> 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
> 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
> be copied. There may be no data in the table.
> Exception in thread "main" java.io.FileNotFoundException: Unable to open
> link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ]
> at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> at
>
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> at
>
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
>
> Basically while exporting to local i dont see .archive directory.why?
>
> Please comment on this -
> It seems we can't export complete snapshot data directly to local file
> system using 'ExportSnapshot' command.
> If we want to copy to outside of cluster first we need to export it to hdfs
> and then use hadoop get command to copy to local file system.
> Is this correct?
>
> Thanks
> -OC
>
>
> On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > The log you pasted was for taking snapshot.
> > Do you have log from ExportSnapshot ?
> >
> > bq. What is the difference between below two commands?
> >
> > This is the code that determines output FileSystem:
> >
> > FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
> >
> > For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of
> > org.apache.hadoop.fs.LocalFileSystem.
> >
> > Cheers
> >
> >
> > On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc...@gmail.com> wrote:
> >
> > > We see same logs for both options
> > >
> > > 013-12-02 09:47:41,311 INFO
> > > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> > FLUSH
> > > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > FileSystem doesn't support getDefaultReplication
> > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > FileSystem doesn't support getDefaultBlockSize
> > > 2013-12-02 09:47:41,337 INFO
> org.apache.hadoop.hbase.procedure.Procedure:
> > > Starting procedure 'tsdb_snap_backup'
> > > 2013-12-02 09:47:41,724 INFO
> org.apache.hadoop.hbase.procedure.Procedure:
> > > Procedure 'tsdb_snap_backup' execution completed
> > > 2013-12-02 09:47:41,724 INFO
> > > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes
> > for
> > > procedure tsdb_snap_backupincluding nodes
> /hbase/online-snapshot/acquired
> > > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > > 2013-12-02 09:47:41,730 INFO
> > > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler:
> Done
> > > waiting - snapshot for tsdb_snap_backup finished!
> > >
> > > It seems we can't export complete snapshot data directly to local file
> > > system using 'ExportSnapshot' command.
> > > If we want to copy to outside of cluster first we need to export it to
> > hdfs
> > > and then use hadoop get command to copy to local file system.
> > > Is this correct?
> > >
> > > What is the difference between below two commands?
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers
> 16;
> > >
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16;
> > >
> > > Thanks
> > > -OC
> > >
> > >
> > >
> > > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Can you pastebin master log during operation #2 ?
> > > >
> > > > There have been at least two fixes since 0.94.10, listed below.
> > > > It would be nice if you can verify this behavior using 0.94.14
> > > >
> > > > Cheers
> > > >
> > > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013)
> | 1
> > > > line
> > > >
> > > > HBASE-8760 possible loss of data in snapshot taken after region split
> > > >
> > ------------------------------------------------------------------------
> > > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013)
> | 1
> > > > line
> > > >
> > > > HBASE-9060 ExportSnapshot job fails if target path contains
> percentage
> > > > character (Jerry He)
> > > >
> > > >
> > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > > >
> > > > > We have created snapshot for all hbase tables and trying to export
> > > > snapshot
> > > > > in two ways.
> > > > >
> > > > > option 1.Export snapshot into same cluster hdfs
> > > > >
> > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers
> 16;
> > > > >
> > > > > Here we are getting full data ( .archive + .hbase-snapshot)
> exported
> > to
> > > > > hdfs:/hbase_backup
> > > > >
> > > > > option 2.Export snapshot to local filesystem
> > > > > command :
> > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup*
> > -mappers
> > > > 16;
> > > > >
> > > > > But with option 2 we only getting .hbase-snapshot exported to local
> > dir
> > > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> > expected
> > > > > behavior or something wrong in option 2.
> > > > >
> > > > > Thanks
> > > > > OC
> > > > >
> > > >
> > >
> >
>
Re: HBase ExportSnapshot
Posted by oc tsdb <oc...@gmail.com>.
here is snapshot export logs.
mastre log:
===========
2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
snapshot export console log:
=============================
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
13/12/02 21:54:24 INFO util.FSVisitor: No families under region
directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
be copied. There may be no data in the table.
13/12/02 21:54:25 INFO util.FSVisitor: No families under region
directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
be copied. There may be no data in the table.
Exception in thread "main" java.io.FileNotFoundException: Unable to open
link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
]
at
org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
at
org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
at
org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
Basically while exporting to local i dont see .archive directory.why?
Please comment on this -
It seems we can't export complete snapshot data directly to local file
system using 'ExportSnapshot' command.
If we want to copy to outside of cluster first we need to export it to hdfs
and then use hadoop get command to copy to local file system.
Is this correct?
Thanks
-OC
On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yu...@gmail.com> wrote:
> The log you pasted was for taking snapshot.
> Do you have log from ExportSnapshot ?
>
> bq. What is the difference between below two commands?
>
> This is the code that determines output FileSystem:
>
> FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
>
> For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of
> org.apache.hadoop.fs.LocalFileSystem.
>
> Cheers
>
>
> On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc...@gmail.com> wrote:
>
> > We see same logs for both options
> >
> > 013-12-02 09:47:41,311 INFO
> > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> FLUSH
> > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > FileSystem doesn't support getDefaultReplication
> > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > FileSystem doesn't support getDefaultBlockSize
> > 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure:
> > Starting procedure 'tsdb_snap_backup'
> > 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure:
> > Procedure 'tsdb_snap_backup' execution completed
> > 2013-12-02 09:47:41,724 INFO
> > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes
> for
> > procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired
> > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > 2013-12-02 09:47:41,730 INFO
> > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done
> > waiting - snapshot for tsdb_snap_backup finished!
> >
> > It seems we can't export complete snapshot data directly to local file
> > system using 'ExportSnapshot' command.
> > If we want to copy to outside of cluster first we need to export it to
> hdfs
> > and then use hadoop get command to copy to local file system.
> > Is this correct?
> >
> > What is the difference between below two commands?
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16;
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16;
> >
> > Thanks
> > -OC
> >
> >
> >
> > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Can you pastebin master log during operation #2 ?
> > >
> > > There have been at least two fixes since 0.94.10, listed below.
> > > It would be nice if you can verify this behavior using 0.94.14
> > >
> > > Cheers
> > >
> > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1
> > > line
> > >
> > > HBASE-8760 possible loss of data in snapshot taken after region split
> > >
> ------------------------------------------------------------------------
> > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1
> > > line
> > >
> > > HBASE-9060 ExportSnapshot job fails if target path contains percentage
> > > character (Jerry He)
> > >
> > >
> > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > >
> > > > We have created snapshot for all hbase tables and trying to export
> > > snapshot
> > > > in two ways.
> > > >
> > > > option 1.Export snapshot into same cluster hdfs
> > > >
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16;
> > > >
> > > > Here we are getting full data ( .archive + .hbase-snapshot) exported
> to
> > > > hdfs:/hbase_backup
> > > >
> > > > option 2.Export snapshot to local filesystem
> > > > command :
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup*
> -mappers
> > > 16;
> > > >
> > > > But with option 2 we only getting .hbase-snapshot exported to local
> dir
> > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> expected
> > > > behavior or something wrong in option 2.
> > > >
> > > > Thanks
> > > > OC
> > > >
> > >
> >
>
Re: HBase ExportSnapshot
Posted by Ted Yu <yu...@gmail.com>.
The log you pasted was for taking snapshot.
Do you have log from ExportSnapshot ?
bq. What is the difference between below two commands?
This is the code that determines output FileSystem:
FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of
org.apache.hadoop.fs.LocalFileSystem.
Cheers
On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc...@gmail.com> wrote:
> We see same logs for both options
>
> 013-12-02 09:47:41,311 INFO
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running FLUSH
> table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> FileSystem doesn't support getDefaultReplication
> 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> FileSystem doesn't support getDefaultBlockSize
> 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure:
> Starting procedure 'tsdb_snap_backup'
> 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure:
> Procedure 'tsdb_snap_backup' execution completed
> 2013-12-02 09:47:41,724 INFO
> org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes for
> procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired
> /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> 2013-12-02 09:47:41,730 INFO
> org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done
> waiting - snapshot for tsdb_snap_backup finished!
>
> It seems we can't export complete snapshot data directly to local file
> system using 'ExportSnapshot' command.
> If we want to copy to outside of cluster first we need to export it to hdfs
> and then use hadoop get command to copy to local file system.
> Is this correct?
>
> What is the difference between below two commands?
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16;
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16;
>
> Thanks
> -OC
>
>
>
> On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Can you pastebin master log during operation #2 ?
> >
> > There have been at least two fixes since 0.94.10, listed below.
> > It would be nice if you can verify this behavior using 0.94.14
> >
> > Cheers
> >
> > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1
> > line
> >
> > HBASE-8760 possible loss of data in snapshot taken after region split
> > ------------------------------------------------------------------------
> > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1
> > line
> >
> > HBASE-9060 ExportSnapshot job fails if target path contains percentage
> > character (Jerry He)
> >
> >
> > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > >
> > > We have created snapshot for all hbase tables and trying to export
> > snapshot
> > > in two ways.
> > >
> > > option 1.Export snapshot into same cluster hdfs
> > >
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16;
> > >
> > > Here we are getting full data ( .archive + .hbase-snapshot) exported to
> > > hdfs:/hbase_backup
> > >
> > > option 2.Export snapshot to local filesystem
> > > command :
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers
> > 16;
> > >
> > > But with option 2 we only getting .hbase-snapshot exported to local dir
> > > (/tmp/hbase_backup) but .archive files are not exported.It is expected
> > > behavior or something wrong in option 2.
> > >
> > > Thanks
> > > OC
> > >
> >
>
Re: HBase ExportSnapshot
Posted by oc tsdb <oc...@gmail.com>.
We see same logs for both options
013-12-02 09:47:41,311 INFO
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running FLUSH
table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
FileSystem doesn't support getDefaultReplication
2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
FileSystem doesn't support getDefaultBlockSize
2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure:
Starting procedure 'tsdb_snap_backup'
2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure:
Procedure 'tsdb_snap_backup' execution completed
2013-12-02 09:47:41,724 INFO
org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes for
procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired
/hbase/online-snapshot/reached /hbase/online-snapshot/abort
2013-12-02 09:47:41,730 INFO
org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done
waiting - snapshot for tsdb_snap_backup finished!
It seems we can't export complete snapshot data directly to local file
system using 'ExportSnapshot' command.
If we want to copy to outside of cluster first we need to export it to hdfs
and then use hadoop get command to copy to local file system.
Is this correct?
What is the difference between below two commands?
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16;
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16;
Thanks
-OC
On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you pastebin master log during operation #2 ?
>
> There have been at least two fixes since 0.94.10, listed below.
> It would be nice if you can verify this behavior using 0.94.14
>
> Cheers
>
> r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1
> line
>
> HBASE-8760 possible loss of data in snapshot taken after region split
> ------------------------------------------------------------------------
> r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1
> line
>
> HBASE-9060 ExportSnapshot job fails if target path contains percentage
> character (Jerry He)
>
>
> On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
>
> > Hi,
> >
> > We have cluster with 4 data nodes and HBase version is 0.94.10.
> >
> > We have created snapshot for all hbase tables and trying to export
> snapshot
> > in two ways.
> >
> > option 1.Export snapshot into same cluster hdfs
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16;
> >
> > Here we are getting full data ( .archive + .hbase-snapshot) exported to
> > hdfs:/hbase_backup
> >
> > option 2.Export snapshot to local filesystem
> > command :
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers
> 16;
> >
> > But with option 2 we only getting .hbase-snapshot exported to local dir
> > (/tmp/hbase_backup) but .archive files are not exported.It is expected
> > behavior or something wrong in option 2.
> >
> > Thanks
> > OC
> >
>
Re: HBase ExportSnapshot
Posted by Ted Yu <yu...@gmail.com>.
Can you pastebin master log during operation #2 ?
There have been at least two fixes since 0.94.10, listed below.
It would be nice if you can verify this behavior using 0.94.14
Cheers
r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1 line
HBASE-8760 possible loss of data in snapshot taken after region split
------------------------------------------------------------------------
r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1 line
HBASE-9060 ExportSnapshot job fails if target path contains percentage
character (Jerry He)
On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> We have cluster with 4 data nodes and HBase version is 0.94.10.
>
> We have created snapshot for all hbase tables and trying to export snapshot
> in two ways.
>
> option 1.Export snapshot into same cluster hdfs
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16;
>
> Here we are getting full data ( .archive + .hbase-snapshot) exported to
> hdfs:/hbase_backup
>
> option 2.Export snapshot to local filesystem
> command :
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers 16;
>
> But with option 2 we only getting .hbase-snapshot exported to local dir
> (/tmp/hbase_backup) but .archive files are not exported.It is expected
> behavior or something wrong in option 2.
>
> Thanks
> OC
>