You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by jaboy <ja...@flytxt.com> on 2016/03/01 07:50:42 UTC
Hbase data migration from version 0.94 to 0.98
Hi,Is this type of migration been successfully performed till now? we have 2
clusters , cluster 1 (with Apache Hadoop 1.0) in which we have data in Hbase
0.94, the other cluster 2 will have Apache Hadoop 2 (with Hbase 0.98). We
need to migrate data from cluster 1 to cluster 2.What should be the best
course of action? we have thought about making cluster 1 data to Hbase 0.98
compatible and then migrating the data to cluster 2.Thanks & Regardsjaboy
-----
Thanks & Regards
Jaboy
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Hbase data migration from version 0.94 to 0.98
Posted by jaboy <ja...@flytxt.com>.
missed one information. The total un-replicated Hbase data in our source
cluster is around 1.5TB.
-----
Thanks & Regards
Jaboy
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172p4078254.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Hbase data migration from version 0.94 to 0.98
Posted by jaboy <ja...@flytxt.com>.
Hi All,
We dont have hbase replication setup till now. I would think that, we would
be doing distcp for the /hbase folder to the destination cluster, hence
would have to do this with a downtime. Not sure, what migration script i
need to run after i have all the data into hadoop 2 (Hbase 0.98) cluster. I
think, i need to make the new hbase recognize all of the migrated table
data. Hence need some help with that, Is their any command or script to do
the same?
-----
Thanks & Regards
Jaboy
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172p4078253.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Hbase data migration from version 0.94 to 0.98
Posted by Samir Ahmic <ah...@gmail.com>.
Thanks for sharing this Rahul, this is quite useful approach for folks who
have replication already configured and can afford downtime during
migration.
Regards
Samir
On Tue, Mar 1, 2016 at 7:26 PM, rahul gidwani <ra...@gmail.com>
wrote:
> Hi Samir,
>
> We did this migration without any downtime. You can look at
> https://issues.apache.org/jira/browse/HBASE-12814.
>
> Basically the steps are as follows:
> 1. Apply HBASE-12814 for 94 branch and 98 branches respectively.
> 2. Rolling restart all regionservers with the thrift replication parameter
> on.
> 3. Add a new replication source to have thrift protocol.
> 4. You can distcp (if you can afford downtime) or you can take a hbase
> snapshot and Export Snapshots if you can't
> 5. pause replication on the source side (no writes to destination)
> 6. upgrade hbase and hadoop
> 7. do the same on the other side
> 8. add a new peer for native replication (protobuf) make sure thrift
> backlog has drained
> 9. Revert HBASE-12814 patches
>
> I know its quite a few steps but we did this successfully with a few large
> clusters 1000+ nodes (upgrading from hadoop-1.x / hbase-94.x -> hadoop-2.x
> / hbase-98.x
>
>
> On Tue, Mar 1, 2016 at 3:33 AM, jaboy <ja...@flytxt.com> wrote:
>
> > Hi Samir,Thanks for sharing the steps you performed. We are trying to
> > accomplish this migration on a production environment. Could you please
> > share the migration script template that worked in your cluster. Not sure
> > if
> > we need to include any special commands in the script other than just
> > 'distcp'. Thanks in advance.
> >
> >
> >
> > -----
> > Thanks & Regards
> > Jaboy
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172p4078176.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
Re: Hbase data migration from version 0.94 to 0.98
Posted by rahul gidwani <ra...@gmail.com>.
Hi Samir,
We did this migration without any downtime. You can look at
https://issues.apache.org/jira/browse/HBASE-12814.
Basically the steps are as follows:
1. Apply HBASE-12814 for 94 branch and 98 branches respectively.
2. Rolling restart all regionservers with the thrift replication parameter
on.
3. Add a new replication source to have thrift protocol.
4. You can distcp (if you can afford downtime) or you can take a hbase
snapshot and Export Snapshots if you can't
5. pause replication on the source side (no writes to destination)
6. upgrade hbase and hadoop
7. do the same on the other side
8. add a new peer for native replication (protobuf) make sure thrift
backlog has drained
9. Revert HBASE-12814 patches
I know its quite a few steps but we did this successfully with a few large
clusters 1000+ nodes (upgrading from hadoop-1.x / hbase-94.x -> hadoop-2.x
/ hbase-98.x
On Tue, Mar 1, 2016 at 3:33 AM, jaboy <ja...@flytxt.com> wrote:
> Hi Samir,Thanks for sharing the steps you performed. We are trying to
> accomplish this migration on a production environment. Could you please
> share the migration script template that worked in your cluster. Not sure
> if
> we need to include any special commands in the script other than just
> 'distcp'. Thanks in advance.
>
>
>
> -----
> Thanks & Regards
> Jaboy
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172p4078176.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: Hbase data migration from version 0.94 to 0.98
Posted by jaboy <ja...@flytxt.com>.
Hi Samir,Thanks for sharing the steps you performed. We are trying to
accomplish this migration on a production environment. Could you please
share the migration script template that worked in your cluster. Not sure if
we need to include any special commands in the script other than just
'distcp'. Thanks in advance.
-----
Thanks & Regards
Jaboy
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172p4078176.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Hbase data migration from version 0.94 to 0.98
Posted by Samir Ahmic <ah...@gmail.com>.
We have recently tested similar migration path source: hadoop1/hbase0.94
--> destination: hadoop2:/hbase0.98. Here is steps that we have performed :
1. stop source hbase cluster
2. use distcp to copy /hbase folder to destination cluster
3. run hbase migrate script on destination cluster (on destination cluster
we have just configured hbase cluster we never start it)
4. start hbase cluster on destination cluster
This approach worked for us. Hope it can help you.
Regards
Samir
On Tue, Mar 1, 2016 at 7:50 AM, jaboy <ja...@flytxt.com> wrote:
> Hi,Is this type of migration been successfully performed till now? we have
> 2
> clusters , cluster 1 (with Apache Hadoop 1.0) in which we have data in
> Hbase
> 0.94, the other cluster 2 will have Apache Hadoop 2 (with Hbase 0.98). We
> need to migrate data from cluster 1 to cluster 2.What should be the best
> course of action? we have thought about making cluster 1 data to Hbase 0.98
> compatible and then migrating the data to cluster 2.Thanks & Regardsjaboy
>
>
>
> -----
> Thanks & Regards
> Jaboy
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Hbase-data-migration-from-version-0-94-to-0-98-tp4078172.html
> Sent from the HBase User mailing list archive at Nabble.com.