You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ted Tuttle <te...@mentacapital.com> on 2012/03/27 01:31:51 UTC
efficient export w/o HDFS/copying
Hello All-
We've been experimenting w/ exporting and restoring our cluster data
from those exports. Our current methodology has the following steps:
* Dump the table from Hbase to HDFS:
o hadoop jar /usr/lib/hbase/hbase-0.92.0.jar export
<tablename> <HDFS location>
* Copy HDFS dump to filesystem:
o hadoop dfs -copyToLocal <Linux Path> <HDFS location>
* Import Linux dump into HDFS with:
o hadoop dfs -copyFromLocal <linux path, inc dumpdir>
<HDFS location>
* Import HDFS data into Hbase:
o hadoop jar /usr/lib/hbase/hbase-0.92.0.jar import
<tablename> <HDFS location>
Is there a method of exporting that skips the HDFS step? We would
ideally like to export from HBase directly to an external filesystem
(e.g. our big slow NAS) skipping the HDFS step.
Any thoughts or links would be appreciated.
-Ted
Re: efficient export w/o HDFS/copying
Posted by Stack <st...@duboce.net>.
On Wed, Mar 28, 2012 at 12:59 AM, Michel Segel
<mi...@hotmail.com> wrote:
> Wouldn't that mean having the NAS attached to all of the nodes in the cluster?
>
Yes. That was the presumption.
St.Ack
Re: efficient export w/o HDFS/copying
Posted by Michel Segel <mi...@hotmail.com>.
Wouldn't that mean having the NAS attached to all of the nodes in the cluster?
Sent from a remote device. Please excuse any typos...
Mike Segel
On Mar 26, 2012, at 11:07 PM, Stack <st...@duboce.net> wrote:
> On Mon, Mar 26, 2012 at 4:31 PM, Ted Tuttle <te...@mentacapital.com> wrote:
>> Is there a method of exporting that skips the HDFS step? We would
>> ideally like to export from HBase directly to an external filesystem
>> (e.g. our big slow NAS) skipping the HDFS step.
>>
>
> Do an OutputFormat that just writes files to your NAS and hook it up
> to the export tool in place of SequenceFileOutputFormat. Set your new
> NASOutputFormat instead of SequenceFileOutputFormat here:
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/Export.html#99
> (You'll probably have to override Exporter to do your customization
> copying bulk of createSubmittableJob into subclass)
>
> St.Ack
>
Re: efficient export w/o HDFS/copying
Posted by Stack <st...@duboce.net>.
On Mon, Mar 26, 2012 at 4:31 PM, Ted Tuttle <te...@mentacapital.com> wrote:
> Is there a method of exporting that skips the HDFS step? We would
> ideally like to export from HBase directly to an external filesystem
> (e.g. our big slow NAS) skipping the HDFS step.
>
Do an OutputFormat that just writes files to your NAS and hook it up
to the export tool in place of SequenceFileOutputFormat. Set your new
NASOutputFormat instead of SequenceFileOutputFormat here:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/Export.html#99
(You'll probably have to override Exporter to do your customization
copying bulk of createSubmittableJob into subclass)
St.Ack