You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Robert Gonzalez <go...@maxpointinteractive.com> on 2011/01/21 22:27:33 UTC
data migration problem
We are migrating from hbase 0.89.20100621+17 (20.4?) to 20.6 and have
encountered some type of compatibility problem. To test the migration, we
deployed 20.6 on a small cluster and copied a table from the
hdfs:/hbase/tablename directory and re-created it using add_table.rb. We first
tried this with 20.4 on the cluster and everything worked, we were able to scan
the data, verify some contents, etc. When we installed 20.6 on it, the
procedure was repeated (add_table.rb etc) but the table did not work. We were
getting 0 for count 'tablename', and scans would not find any data. When we
looked at the hdfs:/hbase/tablename directory, the old regions were there, but
there were new, very short and suspicious "regions" in there with only a
.regioninfo file in them. Is there a compatibility issue between 20.4 and 20.6
data?
Note directory listing trimmed to accomodate 80 char limit of maillist
hbase@c2-m01:/usr/lib/hbase/bin$ hadoop fs -ls /hbase/myurlhash
Found 16 items
/hbase/myurlhash0e72138a329fa0c94a49cef92b680592
/hbase/myurlhash/1442012393
/hbase/myurlhash/1739287575
/hbase/myurlhash/1840351287
/hbase/myurlhash/188268934
/hbase/myurlhash/391140665
/hbase/myurlhash/3cffee491242b1502a6298305dfe3495
/hbase/myurlhash/406321816
/hbase/myurlhash/476529307
/hbase/myurlhash/478436003
/hbase/myurlhash/55f0dc7eb64684d253cce0f49270a0b4
/hbase/myurlhash/6110ff6921f39df866e0f0a4caecc7a6
/hbase/myurlhash/648cdaa84dc24e1ad3277c626c2fefbc
/hbase/myurlhash/9c7d3dceea28e6d385b22970a8d3e516
/hbase/myurlhash/a60536a1ea341ce822efc98fc07106df
/hbase/myurlhash/c8804788fbeb549c1ccfc4c54addc451
--
Robert Gonzalez
Re: data migration problem
Posted by Stack <st...@duboce.net>.
On Fri, Jan 21, 2011 at 1:38 PM, Robert Gonzalez
<go...@maxpointinteractive.com> wrote:
> So does migration to 0.90 eliminate the need for Export/Import? If not, is
> there a page describing this procedure in more detail?
>
See the announcement email: http://su.pr/30inOk. It talks about 'migration'.
St.Ack
Re: data migration problem
Posted by Robert Gonzalez <go...@maxpointinteractive.com>.
Ryan Rawson <ry...@...> writes:
>
> Hi,
>
> Unfortunately the on disk layout of regions in 0.89 and above has
> changed, and 0.20.6 cannot read that.
>
> You should try upgrading to 0.90 if you can!
>
> If you can't, you will need to use the Export and Import map reduce
> jobs to first export the data to sequence files from 0.89, then on the
> 0.20.6 cluster use Import.
>
> Good luck!
> -ryan
So does migration to 0.90 eliminate the need for Export/Import? If not, is
there a page describing this procedure in more detail?
Thanks,
Robert Gonzalez
Re: data migration problem
Posted by Ryan Rawson <ry...@gmail.com>.
You are correct. 0.20 was the OLD stable release, the 0.90 release is
the NEW (as of this week!) and is the preferred release to be using
moving forward.
Note there wont be a 0.20.7, and getting fixes to 0.20.x will be
highly unlikely. Please move to 0.90 and help us keep forward
momentum!
Thanks!
-ryan
On Fri, Jan 21, 2011 at 1:41 PM, Robert Gonzalez
<go...@maxpointinteractive.com> wrote:
> Ryan Rawson <ry...@...> writes:
>
>>
>> Oh and 0.89.... is a major upgrade from 0.20.x... it is NOT 0.20.4.
>>
>> If you are looking to upgrade, 0.90 is it.
>>
>> -ryan
>>
>
>
> The HBase docs seem to indicate that the 0.20.xx versions are more supported
> than the 0.89++ versions, which are described as "development" versions.
>
>
>
>
>
>
>
Re: data migration problem
Posted by Robert Gonzalez <go...@maxpointinteractive.com>.
Ryan Rawson <ry...@...> writes:
>
> Oh and 0.89.... is a major upgrade from 0.20.x... it is NOT 0.20.4.
>
> If you are looking to upgrade, 0.90 is it.
>
> -ryan
>
The HBase docs seem to indicate that the 0.20.xx versions are more supported
than the 0.89++ versions, which are described as "development" versions.
Re: data migration problem
Posted by Ryan Rawson <ry...@gmail.com>.
Oh and 0.89.... is a major upgrade from 0.20.x... it is NOT 0.20.4.
If you are looking to upgrade, 0.90 is it.
-ryan
On Fri, Jan 21, 2011 at 1:33 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Hi,
>
> Unfortunately the on disk layout of regions in 0.89 and above has
> changed, and 0.20.6 cannot read that.
>
> You should try upgrading to 0.90 if you can!
>
> If you can't, you will need to use the Export and Import map reduce
> jobs to first export the data to sequence files from 0.89, then on the
> 0.20.6 cluster use Import.
>
> Good luck!
> -ryan
>
> On Fri, Jan 21, 2011 at 1:27 PM, Robert Gonzalez
> <go...@maxpointinteractive.com> wrote:
>> We are migrating from hbase 0.89.20100621+17 (20.4?) to 20.6 and have
>> encountered some type of compatibility problem. To test the migration, we
>> deployed 20.6 on a small cluster and copied a table from the
>> hdfs:/hbase/tablename directory and re-created it using add_table.rb. We first
>> tried this with 20.4 on the cluster and everything worked, we were able to scan
>> the data, verify some contents, etc. When we installed 20.6 on it, the
>> procedure was repeated (add_table.rb etc) but the table did not work. We were
>> getting 0 for count 'tablename', and scans would not find any data. When we
>> looked at the hdfs:/hbase/tablename directory, the old regions were there, but
>> there were new, very short and suspicious "regions" in there with only a
>> .regioninfo file in them. Is there a compatibility issue between 20.4 and 20.6
>> data?
>>
>> Note directory listing trimmed to accomodate 80 char limit of maillist
>> hbase@c2-m01:/usr/lib/hbase/bin$ hadoop fs -ls /hbase/myurlhash
>> Found 16 items
>> /hbase/myurlhash0e72138a329fa0c94a49cef92b680592
>> /hbase/myurlhash/1442012393
>> /hbase/myurlhash/1739287575
>> /hbase/myurlhash/1840351287
>> /hbase/myurlhash/188268934
>> /hbase/myurlhash/391140665
>> /hbase/myurlhash/3cffee491242b1502a6298305dfe3495
>> /hbase/myurlhash/406321816
>> /hbase/myurlhash/476529307
>> /hbase/myurlhash/478436003
>> /hbase/myurlhash/55f0dc7eb64684d253cce0f49270a0b4
>> /hbase/myurlhash/6110ff6921f39df866e0f0a4caecc7a6
>> /hbase/myurlhash/648cdaa84dc24e1ad3277c626c2fefbc
>> /hbase/myurlhash/9c7d3dceea28e6d385b22970a8d3e516
>> /hbase/myurlhash/a60536a1ea341ce822efc98fc07106df
>> /hbase/myurlhash/c8804788fbeb549c1ccfc4c54addc451
>>
>> --
>> Robert Gonzalez
>>
>>
>>
>
Re: data migration problem
Posted by Ryan Rawson <ry...@gmail.com>.
Hi,
Unfortunately the on disk layout of regions in 0.89 and above has
changed, and 0.20.6 cannot read that.
You should try upgrading to 0.90 if you can!
If you can't, you will need to use the Export and Import map reduce
jobs to first export the data to sequence files from 0.89, then on the
0.20.6 cluster use Import.
Good luck!
-ryan
On Fri, Jan 21, 2011 at 1:27 PM, Robert Gonzalez
<go...@maxpointinteractive.com> wrote:
> We are migrating from hbase 0.89.20100621+17 (20.4?) to 20.6 and have
> encountered some type of compatibility problem. To test the migration, we
> deployed 20.6 on a small cluster and copied a table from the
> hdfs:/hbase/tablename directory and re-created it using add_table.rb. We first
> tried this with 20.4 on the cluster and everything worked, we were able to scan
> the data, verify some contents, etc. When we installed 20.6 on it, the
> procedure was repeated (add_table.rb etc) but the table did not work. We were
> getting 0 for count 'tablename', and scans would not find any data. When we
> looked at the hdfs:/hbase/tablename directory, the old regions were there, but
> there were new, very short and suspicious "regions" in there with only a
> .regioninfo file in them. Is there a compatibility issue between 20.4 and 20.6
> data?
>
> Note directory listing trimmed to accomodate 80 char limit of maillist
> hbase@c2-m01:/usr/lib/hbase/bin$ hadoop fs -ls /hbase/myurlhash
> Found 16 items
> /hbase/myurlhash0e72138a329fa0c94a49cef92b680592
> /hbase/myurlhash/1442012393
> /hbase/myurlhash/1739287575
> /hbase/myurlhash/1840351287
> /hbase/myurlhash/188268934
> /hbase/myurlhash/391140665
> /hbase/myurlhash/3cffee491242b1502a6298305dfe3495
> /hbase/myurlhash/406321816
> /hbase/myurlhash/476529307
> /hbase/myurlhash/478436003
> /hbase/myurlhash/55f0dc7eb64684d253cce0f49270a0b4
> /hbase/myurlhash/6110ff6921f39df866e0f0a4caecc7a6
> /hbase/myurlhash/648cdaa84dc24e1ad3277c626c2fefbc
> /hbase/myurlhash/9c7d3dceea28e6d385b22970a8d3e516
> /hbase/myurlhash/a60536a1ea341ce822efc98fc07106df
> /hbase/myurlhash/c8804788fbeb549c1ccfc4c54addc451
>
> --
> Robert Gonzalez
>
>
>