Using HBase's export/import function...

[Michael Segel <mi...@hotmail.com>]

Ok,

Silly question...

Inside the /usr/lib/hbase/*.jar (base jar for HBase) There's an export/import tool.

If you supply the #versions, and the start time and end time, you can timebox your scan so your map/reduce job will let you do daily, weekly, etc type of incremental backups. 

So here's my questions:


1) Is anyone using this.
2) There isn't any documentation, I'm assuming that the start time and end times are timestamps (long values representing the number of miliseconds since the epoch which are what is being stored in hbase).
3) Is there an easy way to convert a date in to a time stamp? (not in ksh, and I'm struggling on finding a way to reverse the datetime object in python.

Thx

-Mike

 		 	   		  

RE: Using HBase's export/import function...

[Michael Segel <mi...@hotmail.com>]

StAck...

LOL...

The idea is to automate the use of the export function to be run within a cron job. 
(And yes, there are some use cases where we want to actually back data up.. ;-)
I originally wanted to do this in ksh (yeah I'm that old. :-) but ended up looking at Python because I couldn't figure out how to create the time stamp in ksh.

As to documentation... just something which tells us what is meant by start time and end time. (Like that its in ms from the epoch instead of making us assume that.)
[And you know what they say about assumptions.]

As to converting the date / time to a timestamp...

In Python:
You build up a date object then you can do the following:
mytime = datetime.datetime(year,month,day,hour,min,sec) *where hour,min,sec are optional
mytimestamp = time.mktime(mytime.timetuple())

I'm in the process of testing this... I think it will work.

Thx

-Mike




> Date: Fri, 6 Aug 2010 11:28:57 -0700
> Subject: Re: Using HBase's export/import function...
> From: stack@duboce.net
> To: user@hbase.apache.org
> 
> On Fri, Aug 6, 2010 at 11:13 AM, Michael Segel
> <mi...@hotmail.com> wrote:
> > 2) There isn't any documentation, I'm assuming that the start time and end times are timestamps (long values representing the number of miliseconds since the epoch which are what is being stored in hbase).
> 
> Yes.
> 
> What kinda doc. do you need?  The javadoc on the class is minimal:
> http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/Export.html
> 
> 
> > 3) Is there an easy way to convert a date in to a time stamp? (not in ksh, and I'm struggling on finding a way to reverse the datetime object in python.
> >
> 
> On the end of this page it shows you how to do date convertions inside
> in the hbase shell: http://wiki.apache.org/hadoop/Hbase/Shell
> 
> St.Ack
> 
> 
> > Thx
> >
> > -Mike
> >
> >
 		 	   		  

Re: Using HBase's export/import function...

[Stack <st...@duboce.net>]

On Fri, Aug 6, 2010 at 11:13 AM, Michael Segel
<mi...@hotmail.com> wrote:
> 2) There isn't any documentation, I'm assuming that the start time and end times are timestamps (long values representing the number of miliseconds since the epoch which are what is being stored in hbase).

Yes.

What kinda doc. do you need?  The javadoc on the class is minimal:
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/Export.html


> 3) Is there an easy way to convert a date in to a time stamp? (not in ksh, and I'm struggling on finding a way to reverse the datetime object in python.
>

On the end of this page it shows you how to do date convertions inside
in the hbase shell: http://wiki.apache.org/hadoop/Hbase/Shell

St.Ack


> Thx
>
> -Mike
>
>