You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Schad, Bjoern-Bernhard (EXT-Redknee - DE/Berlin)" <bj...@redknee.com> on 2013/06/14 12:00:03 UTC
Migration needed when updating within an Hadoop release
Hi,
has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release?
If so, where could I find information regarding such needed migration?
I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
Thank you very much,
Björn
--
Redknee (Germany) GmbH
Geschäftsleitung / Board of Directors: Chris Newton-Smith
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 139030
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> - But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
You will probably see log messages like 'RPC version mismatch', in this case you have to upgrade the filesystem. If not - all well :)
> - What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
Yes, but not regularly. Please see my next answer.
> - Is there a similar wiki page describing HBase upgrade?
http://hbase.apache.org/upgrading.html
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> - But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
You will probably see log messages like 'RPC version mismatch', in this case you have to upgrade the filesystem. If not - all well :)
> - What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
Yes, but not regularly. Please see my next answer.
> - Is there a similar wiki page describing HBase upgrade?
http://hbase.apache.org/upgrading.html
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> - But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
You will probably see log messages like 'RPC version mismatch', in this case you have to upgrade the filesystem. If not - all well :)
> - What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
Yes, but not regularly. Please see my next answer.
> - Is there a similar wiki page describing HBase upgrade?
http://hbase.apache.org/upgrading.html
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> - But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
You will probably see log messages like 'RPC version mismatch', in this case you have to upgrade the filesystem. If not - all well :)
> - What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
Yes, but not regularly. Please see my next answer.
> - Is there a similar wiki page describing HBase upgrade?
http://hbase.apache.org/upgrading.html
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Migration needed when updating within an Hadoop release
Posted by "Schad, Bjoern-Bernhard (EXT-Redknee - DE/Berlin)" <bj...@redknee.com>.
Hello Alexander,
thanks for your reply. This is very interesting for me indeed.
- But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
Also I have got a similar question regarding HBase. I understand that HBase has its own datamodel on top of/ within Hadoop. It could happen (e.g. between HBase major releases) that this datamodel is changed/ extended/ ...
- What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
- Is there a similar wiki page describing HBase upgrade?
Thank you very much,
Björn
--
Redknee (Germany) GmbH
Geschäftsleitung / Board of Directors: Chris Newton-Smith
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 139030
-----Original Message-----
From: ext Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Friday, June 14, 2013 12:12 PM
To: user@hadoop.apache.org user@hadoop.apache.org
Subject: Re: Migration needed when updating within an Hadoop release
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Migration needed when updating within an Hadoop release
Posted by "Schad, Bjoern-Bernhard (EXT-Redknee - DE/Berlin)" <bj...@redknee.com>.
Hello Alexander,
thanks for your reply. This is very interesting for me indeed.
- But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
Also I have got a similar question regarding HBase. I understand that HBase has its own datamodel on top of/ within Hadoop. It could happen (e.g. between HBase major releases) that this datamodel is changed/ extended/ ...
- What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
- Is there a similar wiki page describing HBase upgrade?
Thank you very much,
Björn
--
Redknee (Germany) GmbH
Geschäftsleitung / Board of Directors: Chris Newton-Smith
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 139030
-----Original Message-----
From: ext Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Friday, June 14, 2013 12:12 PM
To: user@hadoop.apache.org user@hadoop.apache.org
Subject: Re: Migration needed when updating within an Hadoop release
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Migration needed when updating within an Hadoop release
Posted by "Schad, Bjoern-Bernhard (EXT-Redknee - DE/Berlin)" <bj...@redknee.com>.
Hello Alexander,
thanks for your reply. This is very interesting for me indeed.
- But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
Also I have got a similar question regarding HBase. I understand that HBase has its own datamodel on top of/ within Hadoop. It could happen (e.g. between HBase major releases) that this datamodel is changed/ extended/ ...
- What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
- Is there a similar wiki page describing HBase upgrade?
Thank you very much,
Björn
--
Redknee (Germany) GmbH
Geschäftsleitung / Board of Directors: Chris Newton-Smith
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 139030
-----Original Message-----
From: ext Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Friday, June 14, 2013 12:12 PM
To: user@hadoop.apache.org user@hadoop.apache.org
Subject: Re: Migration needed when updating within an Hadoop release
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Migration needed when updating within an Hadoop release
Posted by "Schad, Bjoern-Bernhard (EXT-Redknee - DE/Berlin)" <bj...@redknee.com>.
Hello Alexander,
thanks for your reply. This is very interesting for me indeed.
- But what about minor updates, e. g. from 1.0.1 to 1.0.4? Has this ever happened for such updates?
Also I have got a similar question regarding HBase. I understand that HBase has its own datamodel on top of/ within Hadoop. It could happen (e.g. between HBase major releases) that this datamodel is changed/ extended/ ...
- What about HBase minor releases in this context? Have data model changes/ conversions/ migrations ever happened between minor releases?
- Is there a similar wiki page describing HBase upgrade?
Thank you very much,
Björn
--
Redknee (Germany) GmbH
Geschäftsleitung / Board of Directors: Chris Newton-Smith
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 139030
-----Original Message-----
From: ext Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Friday, June 14, 2013 12:12 PM
To: user@hadoop.apache.org user@hadoop.apache.org
Subject: Re: Migration needed when updating within an Hadoop release
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Excuse the typo, should be :
Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz <wg...@gmail.com> wrote:
> Hi Björn,
>
>> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
>
> Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
>
>> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
> Depends how much data you've stored. Michael has written a excellent blog post about:
> http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
>
> Cheers,
> Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
Depends how much data you've stored. Michael has written a excellent blog post about:
http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
Depends how much data you've stored. Michael has written a excellent blog post about:
http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
Depends how much data you've stored. Michael has written a excellent blog post about:
http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Migration needed when updating within an Hadoop release
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Björn,
> has it ever happened that a migration of persistent data has been needed (or automatically executed) when updating a Hadoop installation within a release? If so, where could I find information regarding such needed migration?
Normally, when you change the minor release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches.
> I would be interested because the runtime of such migration would probably depend on the amount of managed data and had to be planned.
Depends how much data you've stored. Michael has written a excellent blog post about:
http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/
Cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF