You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hari Sreekumar <hs...@clickable.com> on 2010/11/18 05:25:10 UTC
Restoring table from HFiles
Hi,
I just wanted to know if it is possible to copy an HBase table into
another HDFS by simply copying the <tablename> directory from old HDFS to
local system and dumping it into it into new HDFS?
thanks,
hari
Re: Restoring table from HFiles
Posted by Hari Sreekumar <hs...@clickable.com>.
ah, that's what I was wondering, thanks for the information everyone!
hari
On Fri, Nov 19, 2010 at 12:08 AM, Lars George <la...@gmail.com> wrote:
> No and pretty much no.
>
> You cannot simple throw tables together as that would stuff up region
> boundaries most likely. I would only replace or add a new table that
> way (if at all). And the add_table.rb is more of a kludge as this was
> needed to fix sick clusters in the past. This is not something you
> want to do in production unless you know what you are doing. Then
> rather go Ted's route, i.e. use the Export/Import helpers.
>
> Lars
>
> On Thu, Nov 18, 2010 at 5:58 PM, Hari Sreekumar
> <hs...@clickable.com> wrote:
> > Okay.. so add_table can be run even if there is an existing table by the
> > same name on the new DFS? Can you give me a basic idea of how add_table
> > works/what it really does inside? I am basically interested in dumping
> > processed HBase data into backup servers instead of dumping raw data.
> That
> > would save me a lot of space and headache in the event of a system crash
> > etc. Is add_table reliable for this?
> >
> > thanks,
> > hari
> >
> > On Thu, Nov 18, 2010 at 10:08 PM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> I would say "yes", conditionally.
> >>
> >> But indeed you have to use add_table.rb to add the copied over regions
> to
> >> the META region of the target cluster.
> >>
> >> And of course if you copy over table data as HFiles you have to at least
> >> disable the table on the source cluster or shut it down before the copy,
> so
> >> to be sure all pending writes are flushed from memstore to disk and that
> >> splits and compactions do not run while you are copying.
> Splits/compactions
> >> change hfile layout on disk. You must freeze the table so your copy is
> >> consistent (source data remains unchanged start to finish).
> >>
> >> Best regards,
> >>
> >> - Andy
> >>
> >>
> >> --- On Wed, 11/17/10, Lars George <la...@gmail.com> wrote:
> >>
> >> > From: Lars George <la...@gmail.com>
> >> > Subject: Re: Restoring table from HFiles
> >> > To: user@hbase.apache.org
> >> > Date: Wednesday, November 17, 2010, 11:51 PM
> >> > I would not say "no" immediately. I
> >> > know some have done so (given the
> >> > version was the same) and used add_table.rb to add the
> >> > table to META.
> >> > YMMV.
> >> >
> >> > Lars
> >> >
> >> > On Thu, Nov 18, 2010 at 6:01 AM, Ted Yu <yu...@gmail.com>
> >> > wrote:
> >> > > No.
> >> > >
> >> > > See https://issues.apache.org/jira/browse/HBASE-1684
> >> > >
> >> > > On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <
> >> hsreekumar@clickable.com>wrote:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> I just wanted to know if it is possible to
> >> > copy an HBase table into
> >> > >> another HDFS by simply copying the
> >> > <tablename> directory from old HDFS to
> >> > >> local system and dumping it into it into new
> >> > HDFS?
> >> > >>
> >> > >> thanks,
> >> > >> hari
> >> > >>
> >> > >
> >> >
> >>
> >>
> >>
> >>
> >
>
Re: Restoring table from HFiles
Posted by Lars George <la...@gmail.com>.
No and pretty much no.
You cannot simple throw tables together as that would stuff up region
boundaries most likely. I would only replace or add a new table that
way (if at all). And the add_table.rb is more of a kludge as this was
needed to fix sick clusters in the past. This is not something you
want to do in production unless you know what you are doing. Then
rather go Ted's route, i.e. use the Export/Import helpers.
Lars
On Thu, Nov 18, 2010 at 5:58 PM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Okay.. so add_table can be run even if there is an existing table by the
> same name on the new DFS? Can you give me a basic idea of how add_table
> works/what it really does inside? I am basically interested in dumping
> processed HBase data into backup servers instead of dumping raw data. That
> would save me a lot of space and headache in the event of a system crash
> etc. Is add_table reliable for this?
>
> thanks,
> hari
>
> On Thu, Nov 18, 2010 at 10:08 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> I would say "yes", conditionally.
>>
>> But indeed you have to use add_table.rb to add the copied over regions to
>> the META region of the target cluster.
>>
>> And of course if you copy over table data as HFiles you have to at least
>> disable the table on the source cluster or shut it down before the copy, so
>> to be sure all pending writes are flushed from memstore to disk and that
>> splits and compactions do not run while you are copying. Splits/compactions
>> change hfile layout on disk. You must freeze the table so your copy is
>> consistent (source data remains unchanged start to finish).
>>
>> Best regards,
>>
>> - Andy
>>
>>
>> --- On Wed, 11/17/10, Lars George <la...@gmail.com> wrote:
>>
>> > From: Lars George <la...@gmail.com>
>> > Subject: Re: Restoring table from HFiles
>> > To: user@hbase.apache.org
>> > Date: Wednesday, November 17, 2010, 11:51 PM
>> > I would not say "no" immediately. I
>> > know some have done so (given the
>> > version was the same) and used add_table.rb to add the
>> > table to META.
>> > YMMV.
>> >
>> > Lars
>> >
>> > On Thu, Nov 18, 2010 at 6:01 AM, Ted Yu <yu...@gmail.com>
>> > wrote:
>> > > No.
>> > >
>> > > See https://issues.apache.org/jira/browse/HBASE-1684
>> > >
>> > > On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <
>> hsreekumar@clickable.com>wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> I just wanted to know if it is possible to
>> > copy an HBase table into
>> > >> another HDFS by simply copying the
>> > <tablename> directory from old HDFS to
>> > >> local system and dumping it into it into new
>> > HDFS?
>> > >>
>> > >> thanks,
>> > >> hari
>> > >>
>> > >
>> >
>>
>>
>>
>>
>
Re: Restoring table from HFiles
Posted by Hari Sreekumar <hs...@clickable.com>.
Okay.. so add_table can be run even if there is an existing table by the
same name on the new DFS? Can you give me a basic idea of how add_table
works/what it really does inside? I am basically interested in dumping
processed HBase data into backup servers instead of dumping raw data. That
would save me a lot of space and headache in the event of a system crash
etc. Is add_table reliable for this?
thanks,
hari
On Thu, Nov 18, 2010 at 10:08 PM, Andrew Purtell <ap...@apache.org>wrote:
> I would say "yes", conditionally.
>
> But indeed you have to use add_table.rb to add the copied over regions to
> the META region of the target cluster.
>
> And of course if you copy over table data as HFiles you have to at least
> disable the table on the source cluster or shut it down before the copy, so
> to be sure all pending writes are flushed from memstore to disk and that
> splits and compactions do not run while you are copying. Splits/compactions
> change hfile layout on disk. You must freeze the table so your copy is
> consistent (source data remains unchanged start to finish).
>
> Best regards,
>
> - Andy
>
>
> --- On Wed, 11/17/10, Lars George <la...@gmail.com> wrote:
>
> > From: Lars George <la...@gmail.com>
> > Subject: Re: Restoring table from HFiles
> > To: user@hbase.apache.org
> > Date: Wednesday, November 17, 2010, 11:51 PM
> > I would not say "no" immediately. I
> > know some have done so (given the
> > version was the same) and used add_table.rb to add the
> > table to META.
> > YMMV.
> >
> > Lars
> >
> > On Thu, Nov 18, 2010 at 6:01 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > No.
> > >
> > > See https://issues.apache.org/jira/browse/HBASE-1684
> > >
> > > On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <
> hsreekumar@clickable.com>wrote:
> > >
> > >> Hi,
> > >>
> > >> I just wanted to know if it is possible to
> > copy an HBase table into
> > >> another HDFS by simply copying the
> > <tablename> directory from old HDFS to
> > >> local system and dumping it into it into new
> > HDFS?
> > >>
> > >> thanks,
> > >> hari
> > >>
> > >
> >
>
>
>
>
Re: Restoring table from HFiles
Posted by Andrew Purtell <ap...@apache.org>.
I would say "yes", conditionally.
But indeed you have to use add_table.rb to add the copied over regions to the META region of the target cluster.
And of course if you copy over table data as HFiles you have to at least disable the table on the source cluster or shut it down before the copy, so to be sure all pending writes are flushed from memstore to disk and that splits and compactions do not run while you are copying. Splits/compactions change hfile layout on disk. You must freeze the table so your copy is consistent (source data remains unchanged start to finish).
Best regards,
- Andy
--- On Wed, 11/17/10, Lars George <la...@gmail.com> wrote:
> From: Lars George <la...@gmail.com>
> Subject: Re: Restoring table from HFiles
> To: user@hbase.apache.org
> Date: Wednesday, November 17, 2010, 11:51 PM
> I would not say "no" immediately. I
> know some have done so (given the
> version was the same) and used add_table.rb to add the
> table to META.
> YMMV.
>
> Lars
>
> On Thu, Nov 18, 2010 at 6:01 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > No.
> >
> > See https://issues.apache.org/jira/browse/HBASE-1684
> >
> > On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <hs...@clickable.com>wrote:
> >
> >> Hi,
> >>
> >> I just wanted to know if it is possible to
> copy an HBase table into
> >> another HDFS by simply copying the
> <tablename> directory from old HDFS to
> >> local system and dumping it into it into new
> HDFS?
> >>
> >> thanks,
> >> hari
> >>
> >
>
Re: Restoring table from HFiles
Posted by Lars George <la...@gmail.com>.
I would not say "no" immediately. I know some have done so (given the
version was the same) and used add_table.rb to add the table to META.
YMMV.
Lars
On Thu, Nov 18, 2010 at 6:01 AM, Ted Yu <yu...@gmail.com> wrote:
> No.
>
> See https://issues.apache.org/jira/browse/HBASE-1684
>
> On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <hs...@clickable.com>wrote:
>
>> Hi,
>>
>> I just wanted to know if it is possible to copy an HBase table into
>> another HDFS by simply copying the <tablename> directory from old HDFS to
>> local system and dumping it into it into new HDFS?
>>
>> thanks,
>> hari
>>
>
Re: Restoring table from HFiles
Posted by Ted Yu <yu...@gmail.com>.
No.
See https://issues.apache.org/jira/browse/HBASE-1684
On Wed, Nov 17, 2010 at 8:25 PM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hi,
>
> I just wanted to know if it is possible to copy an HBase table into
> another HDFS by simply copying the <tablename> directory from old HDFS to
> local system and dumping it into it into new HDFS?
>
> thanks,
> hari
>