You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 陈加俊 <cj...@gmail.com> on 2011/03/09 11:27:32 UTC
How to copy HTable from one cluster to another cluster ?
How to copy HTable from one cluster to another cluster ?
The table is very big .
--
Thanks & Best regards
jiajun
RE: How to copy HTable from one cluster to another cluster ?
Posted by "Buttler, David" <bu...@llnl.gov>.
You will have to make sure you are not writing to the table that you are copying to the local disk.
It seems reasonable to me, but I would suggest trying it out with a small data set to make sure you get the process down.
Dave
-----Original Message-----
From: 陈加俊 [mailto:cjjvictory@gmail.com]
Sent: Wednesday, March 30, 2011 6:48 PM
To: user@hbase.apache.org
Cc: Jean-Daniel Cryans
Subject: Re: How to copy HTable from one cluster to another cluster ?
Step by follows ,Is it ok?
1. bin/hadoop fs -copyToLocal
2. bin/hadoop fs -copyFromLocal
3. bin/hbase org.jruby.Main bin/add_table.rb
Any other solution ?
2011/3/31 Jean-Daniel Cryans <jd...@apache.org>
> Put your data on hard drives, ship them to the other data center?
>
> On Wed, Mar 30, 2011 at 12:44 AM, 陈加俊 <cj...@gmail.com> wrote:
> > My cluster 1 is in shanghai and cluster 2 is in beijing. So the cluster 1
> > can't see the cluster 2.
> >
> > any hints?
> >
> > 2011/3/10 shixing <pa...@gmail.com>
> >
> >> Because the distcp's per map runs long time than the
> mapred.task.timeout,
> >> so
> >> you can change the value in the hadoop-site.xml to a large num in
> >> milliseconds.
> >>
> >> On Thu, Mar 10, 2011 at 1:14 PM, 陈加俊 <cj...@gmail.com> wrote:
> >>
> >> > Thank you I used HBase-0.20.6.
> >> >
> >> > On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com>
> >> wrote:
> >> >
> >> > > What HBase version are you on?
> >> > > If you are on 0.90+ there is a Copy Table MR job that would do this
> for
> >> > > you.
> >> > >
> >> > >
> >> >
> >>
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
> >> > > --Suraj
> >> > >
> >> > > On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
> >> > >
> >> > > > How to copy HTable from one cluster to another cluster ?
> >> > > >
> >> > > > The table is very big .
> >> > > >
> >> > > > --
> >> > > > Thanks & Best regards
> >> > > > jiajun
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks & Best regards
> >> > jiajun
> >> >
> >>
> >>
> >>
> >> --
> >> Best wishes!
> >> My Friend~
> >>
> >
> >
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
Step by follows ,Is it ok?
1. bin/hadoop fs -copyToLocal
2. bin/hadoop fs -copyFromLocal
3. bin/hbase org.jruby.Main bin/add_table.rb
Any other solution ?
2011/3/31 Jean-Daniel Cryans <jd...@apache.org>
> Put your data on hard drives, ship them to the other data center?
>
> On Wed, Mar 30, 2011 at 12:44 AM, 陈加俊 <cj...@gmail.com> wrote:
> > My cluster 1 is in shanghai and cluster 2 is in beijing. So the cluster 1
> > can't see the cluster 2.
> >
> > any hints?
> >
> > 2011/3/10 shixing <pa...@gmail.com>
> >
> >> Because the distcp's per map runs long time than the
> mapred.task.timeout,
> >> so
> >> you can change the value in the hadoop-site.xml to a large num in
> >> milliseconds.
> >>
> >> On Thu, Mar 10, 2011 at 1:14 PM, 陈加俊 <cj...@gmail.com> wrote:
> >>
> >> > Thank you I used HBase-0.20.6.
> >> >
> >> > On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com>
> >> wrote:
> >> >
> >> > > What HBase version are you on?
> >> > > If you are on 0.90+ there is a Copy Table MR job that would do this
> for
> >> > > you.
> >> > >
> >> > >
> >> >
> >>
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
> >> > > --Suraj
> >> > >
> >> > > On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
> >> > >
> >> > > > How to copy HTable from one cluster to another cluster ?
> >> > > >
> >> > > > The table is very big .
> >> > > >
> >> > > > --
> >> > > > Thanks & Best regards
> >> > > > jiajun
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks & Best regards
> >> > jiajun
> >> >
> >>
> >>
> >>
> >> --
> >> Best wishes!
> >> My Friend~
> >>
> >
> >
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Put your data on hard drives, ship them to the other data center?
On Wed, Mar 30, 2011 at 12:44 AM, 陈加俊 <cj...@gmail.com> wrote:
> My cluster 1 is in shanghai and cluster 2 is in beijing. So the cluster 1
> can't see the cluster 2.
>
> any hints?
>
> 2011/3/10 shixing <pa...@gmail.com>
>
>> Because the distcp's per map runs long time than the mapred.task.timeout,
>> so
>> you can change the value in the hadoop-site.xml to a large num in
>> milliseconds.
>>
>> On Thu, Mar 10, 2011 at 1:14 PM, 陈加俊 <cj...@gmail.com> wrote:
>>
>> > Thank you I used HBase-0.20.6.
>> >
>> > On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com>
>> wrote:
>> >
>> > > What HBase version are you on?
>> > > If you are on 0.90+ there is a Copy Table MR job that would do this for
>> > > you.
>> > >
>> > >
>> >
>> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
>> > > --Suraj
>> > >
>> > > On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
>> > >
>> > > > How to copy HTable from one cluster to another cluster ?
>> > > >
>> > > > The table is very big .
>> > > >
>> > > > --
>> > > > Thanks & Best regards
>> > > > jiajun
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks & Best regards
>> > jiajun
>> >
>>
>>
>>
>> --
>> Best wishes!
>> My Friend~
>>
>
>
>
> --
> Thanks & Best regards
> jiajun
>
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
My cluster 1 is in shanghai and cluster 2 is in beijing. So the cluster 1
can't see the cluster 2.
any hints?
2011/3/10 shixing <pa...@gmail.com>
> Because the distcp's per map runs long time than the mapred.task.timeout,
> so
> you can change the value in the hadoop-site.xml to a large num in
> milliseconds.
>
> On Thu, Mar 10, 2011 at 1:14 PM, 陈加俊 <cj...@gmail.com> wrote:
>
> > Thank you I used HBase-0.20.6.
> >
> > On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com>
> wrote:
> >
> > > What HBase version are you on?
> > > If you are on 0.90+ there is a Copy Table MR job that would do this for
> > > you.
> > >
> > >
> >
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
> > > --Suraj
> > >
> > > On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
> > >
> > > > How to copy HTable from one cluster to another cluster ?
> > > >
> > > > The table is very big .
> > > >
> > > > --
> > > > Thanks & Best regards
> > > > jiajun
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
>
>
> --
> Best wishes!
> My Friend~
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by shixing <pa...@gmail.com>.
Because the distcp's per map runs long time than the mapred.task.timeout, so
you can change the value in the hadoop-site.xml to a large num in
milliseconds.
On Thu, Mar 10, 2011 at 1:14 PM, 陈加俊 <cj...@gmail.com> wrote:
> Thank you I used HBase-0.20.6.
>
> On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com> wrote:
>
> > What HBase version are you on?
> > If you are on 0.90+ there is a Copy Table MR job that would do this for
> > you.
> >
> >
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
> > --Suraj
> >
> > On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
> >
> > > How to copy HTable from one cluster to another cluster ?
> > >
> > > The table is very big .
> > >
> > > --
> > > Thanks & Best regards
> > > jiajun
> > >
> >
>
>
>
> --
> Thanks & Best regards
> jiajun
>
--
Best wishes!
My Friend~
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
Thank you I used HBase-0.20.6.
On Thu, Mar 10, 2011 at 1:26 AM, Suraj Varma <sv...@gmail.com> wrote:
> What HBase version are you on?
> If you are on 0.90+ there is a Copy Table MR job that would do this for
> you.
>
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
> --Suraj
>
> On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
>
> > How to copy HTable from one cluster to another cluster ?
> >
> > The table is very big .
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by Suraj Varma <sv...@gmail.com>.
What HBase version are you on?
If you are on 0.90+ there is a Copy Table MR job that would do this for you.
http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/CopyTable.html
--Suraj
On Wed, Mar 9, 2011 at 2:27 AM, 陈加俊 <cj...@gmail.com> wrote:
> How to copy HTable from one cluster to another cluster ?
>
> The table is very big .
>
> --
> Thanks & Best regards
> jiajun
>
Re: How to copy HTable from one cluster to another cluster ?
Posted by Stack <st...@duboce.net>.
An adornment to Shixing's answer is Xavier's blog on a big move done
recently by Mozilla crew:
http://blog.mozilla.com/data/2011/02/04/migrating-hbase-in-the-trenches/
St.Ack
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
/app/cloud/hadoop/bin/hadoop distcp hdfs://
master.uc.uuwatch.com:9000/hbase/cjjSite hdfs://192.168.0.181:9000/hba/e/
11/03/10 11:42:04 INFO tools.DistCp: srcPaths=[hdfs://
master.uc.uuwatch.com:9000/hbase/cjjSite]
11/03/10 11:42:04 INFO tools.DistCp: destPath=hdfs://
192.168.0.181:9000/hbase
11/03/10 11:52:34 INFO tools.DistCp: srcCount=21
11/03/10 12:08:44 INFO mapred.JobClient: Running job: job_201012131904_0004
11/03/10 12:08:48 INFO mapred.JobClient: map 0% reduce 0%
11/03/10 12:18:52 INFO mapred.JobClient: Task Id :
attempt_201012131904_0004_m_000002_0, Status : FAILED
Task attempt_201012131904_0004_m_000002_0 failed to report status for 602
seconds. Killing!
11/03/10 12:28:56 INFO mapred.JobClient: Task Id :
attempt_201012131904_0004_r_000001_0, Status : FAILED
Task attempt_201012131904_0004_r_000001_0 failed to report status for 601
seconds. Killing!
On Thu, Mar 10, 2011 at 11:38 AM, 陈加俊 <cj...@gmail.com> wrote:
> /app/cloud/hadoop/bin/hadoop distcp hdfs://
> master.uc.uuwatch.com:9000/hbase/cjjProgramme hdfs://
> 192.168.0.181:9000/hbase/
> 11/03/10 11:36:24 INFO tools.DistCp: srcPaths=[hdfs://
> master.uc.uuwatch.com:9000/hbase/cjjProgramme]
> 11/03/10 11:36:24 INFO tools.DistCp: destPath=hdfs://
> 192.168.0.181:9000/hbase
> 11/03/10 11:36:27 INFO tools.DistCp: srcCount=3180
> 11/03/10 11:36:28 INFO mapred.JobClient: Running job: job_201012131904_0001
> 11/03/10 11:36:29 INFO mapred.JobClient: map 0% reduce 0%
>
> and master.uc.uuwatch.com -> 192.168.5.151
>
> On Thu, Mar 10, 2011 at 11:23 AM, 陈加俊 <cj...@gmail.com> wrote:
>
>> The two cluster is in diffrent LAN , but one PC has two network card so
>> could connect the two LAN.
>>
>> Any hints?
>>
>>
>> On Wed, Mar 9, 2011 at 8:05 PM, shixing <pa...@gmail.com> wrote:
>>
>>> 1.You use hdfs-distcp the table files to the destination.
>>> for example:
>>>
>>> distcp hdpnn1:/hbase/TABLE_NAME to hdpnn2:/hbase/TABLE_NAME
>>>
>>> 2. Import the meta data from hbase1 '.META.' table where the row's prefix
>>> is
>>> TABLE_NAME to hbase2 '.META.'
>>>
>>>
>>>
>>> On Wed, Mar 9, 2011 at 6:27 PM, 陈加俊 <cj...@gmail.com> wrote:
>>>
>>> > How to copy HTable from one cluster to another cluster ?
>>> >
>>> > The table is very big .
>>> >
>>> > --
>>> > Thanks & Best regards
>>> > jiajun
>>> >
>>>
>>>
>>>
>>> --
>>> Best wishes!
>>> My Friend~
>>>
>>
>>
>>
>> --
>> Thanks & Best regards
>> jiajun
>>
>>
>
>
> --
> Thanks & Best regards
> jiajun
>
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
/app/cloud/hadoop/bin/hadoop distcp hdfs://
master.uc.uuwatch.com:9000/hbase/cjjProgramme hdfs://
192.168.0.181:9000/hbase/
11/03/10 11:36:24 INFO tools.DistCp: srcPaths=[hdfs://
master.uc.uuwatch.com:9000/hbase/cjjProgramme]
11/03/10 11:36:24 INFO tools.DistCp: destPath=hdfs://
192.168.0.181:9000/hbase
11/03/10 11:36:27 INFO tools.DistCp: srcCount=3180
11/03/10 11:36:28 INFO mapred.JobClient: Running job: job_201012131904_0001
11/03/10 11:36:29 INFO mapred.JobClient: map 0% reduce 0%
and master.uc.uuwatch.com -> 192.168.5.151
On Thu, Mar 10, 2011 at 11:23 AM, 陈加俊 <cj...@gmail.com> wrote:
> The two cluster is in diffrent LAN , but one PC has two network card so
> could connect the two LAN.
>
> Any hints?
>
>
> On Wed, Mar 9, 2011 at 8:05 PM, shixing <pa...@gmail.com> wrote:
>
>> 1.You use hdfs-distcp the table files to the destination.
>> for example:
>>
>> distcp hdpnn1:/hbase/TABLE_NAME to hdpnn2:/hbase/TABLE_NAME
>>
>> 2. Import the meta data from hbase1 '.META.' table where the row's prefix
>> is
>> TABLE_NAME to hbase2 '.META.'
>>
>>
>>
>> On Wed, Mar 9, 2011 at 6:27 PM, 陈加俊 <cj...@gmail.com> wrote:
>>
>> > How to copy HTable from one cluster to another cluster ?
>> >
>> > The table is very big .
>> >
>> > --
>> > Thanks & Best regards
>> > jiajun
>> >
>>
>>
>>
>> --
>> Best wishes!
>> My Friend~
>>
>
>
>
> --
> Thanks & Best regards
> jiajun
>
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by 陈加俊 <cj...@gmail.com>.
The two cluster is in diffrent LAN , but one PC has two network card so
could connect the two LAN.
Any hints?
On Wed, Mar 9, 2011 at 8:05 PM, shixing <pa...@gmail.com> wrote:
> 1.You use hdfs-distcp the table files to the destination.
> for example:
>
> distcp hdpnn1:/hbase/TABLE_NAME to hdpnn2:/hbase/TABLE_NAME
>
> 2. Import the meta data from hbase1 '.META.' table where the row's prefix
> is
> TABLE_NAME to hbase2 '.META.'
>
>
>
> On Wed, Mar 9, 2011 at 6:27 PM, 陈加俊 <cj...@gmail.com> wrote:
>
> > How to copy HTable from one cluster to another cluster ?
> >
> > The table is very big .
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
>
>
> --
> Best wishes!
> My Friend~
>
--
Thanks & Best regards
jiajun
Re: How to copy HTable from one cluster to another cluster ?
Posted by shixing <pa...@gmail.com>.
1.You use hdfs-distcp the table files to the destination.
for example:
distcp hdpnn1:/hbase/TABLE_NAME to hdpnn2:/hbase/TABLE_NAME
2. Import the meta data from hbase1 '.META.' table where the row's prefix is
TABLE_NAME to hbase2 '.META.'
On Wed, Mar 9, 2011 at 6:27 PM, 陈加俊 <cj...@gmail.com> wrote:
> How to copy HTable from one cluster to another cluster ?
>
> The table is very big .
>
> --
> Thanks & Best regards
> jiajun
>
--
Best wishes!
My Friend~