You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ian Brooks <i....@sensewhere.com> on 2014/05/30 11:54:26 UTC
Upgrade to 2.4
Hi,
I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades would be useful to us. Unless i'm missing something, I can't see how to upgrade from 2.3 to 2.4
-rollingUpgrade doesnt work as the old namenode doesn't understand the command and -upgrade is no longer available.
I had a quick google and can't find any documentation on this upgrade, does anyone know how its supposed to be done?
-Ian
Re: Upgrade to 2.4
Posted by Ian Brooks <i....@sensewhere.com>.
Thanks,
Ill give it a try.
-Ian
On Friday 30 May 2014 19:43:00 Zesheng Wu wrote:
> Hi Ian,
> -rollingUpgrade is available since hadoop 2.4, so you can't use
> -rollingUpgrade to upgrade a 2.3 cluster to 2.4.
> Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
> whole cluster and then upgrade the cluster.
> I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
> are some details, wish this can give some help to you:
>
> 1. Put the active NN into safemode: dfsadmin -safemode enter
> 2. Perform a saveNamespace operation: dfsadmin -saveNamespace
> 3. For each component you are using, back up configuration data, databases,
> and other important files
> 4. Shutdown hadoop serverice across your entire cluster
> 5. Check each host to make sure that there are no processes running
> hdfs/yarn
> 6. Back up your HDFS metadata
> 7. Upgrade and start the Journal Nodes and ZKFC
> 8. Make sure that all the journal nodes and zkfc have started normally
> 9. Execute this upgrade operation on active NN: namenode -upgrade
> (Important)
> 10. Wait and make sure from ANN's log that upgrade is completed (from log)
> 11. Bootstrap standby NN: namenode -bootstrapStandby
> 12. Start standby NN: namenode start
> 13. Start each of the DNs: datanode start
> 14. Make sure that everything is running smoothly, this could take a matter
> of days, or even weeks
> 15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
>
>
> 2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
>
> > Hi,
> >
> > I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> > would be useful to us. Unless i'm missing something, I can't see how to
> > upgrade from 2.3 to 2.4
> >
> > -rollingUpgrade doesnt work as the old namenode doesn't understand the
> > command and -upgrade is no longer available.
> >
> > I had a quick google and can't find any documentation on this upgrade,
> > does anyone know how its supposed to be done?
> >
> > -Ian
> >
>
>
>
>
--
-Ian Brooks
Senior server administrator - Sensewhere
Re: Upgrade to 2.4
Posted by Ian Brooks <i....@sensewhere.com>.
Thanks,
Ill give it a try.
-Ian
On Friday 30 May 2014 19:43:00 Zesheng Wu wrote:
> Hi Ian,
> -rollingUpgrade is available since hadoop 2.4, so you can't use
> -rollingUpgrade to upgrade a 2.3 cluster to 2.4.
> Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
> whole cluster and then upgrade the cluster.
> I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
> are some details, wish this can give some help to you:
>
> 1. Put the active NN into safemode: dfsadmin -safemode enter
> 2. Perform a saveNamespace operation: dfsadmin -saveNamespace
> 3. For each component you are using, back up configuration data, databases,
> and other important files
> 4. Shutdown hadoop serverice across your entire cluster
> 5. Check each host to make sure that there are no processes running
> hdfs/yarn
> 6. Back up your HDFS metadata
> 7. Upgrade and start the Journal Nodes and ZKFC
> 8. Make sure that all the journal nodes and zkfc have started normally
> 9. Execute this upgrade operation on active NN: namenode -upgrade
> (Important)
> 10. Wait and make sure from ANN's log that upgrade is completed (from log)
> 11. Bootstrap standby NN: namenode -bootstrapStandby
> 12. Start standby NN: namenode start
> 13. Start each of the DNs: datanode start
> 14. Make sure that everything is running smoothly, this could take a matter
> of days, or even weeks
> 15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
>
>
> 2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
>
> > Hi,
> >
> > I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> > would be useful to us. Unless i'm missing something, I can't see how to
> > upgrade from 2.3 to 2.4
> >
> > -rollingUpgrade doesnt work as the old namenode doesn't understand the
> > command and -upgrade is no longer available.
> >
> > I had a quick google and can't find any documentation on this upgrade,
> > does anyone know how its supposed to be done?
> >
> > -Ian
> >
>
>
>
>
--
-Ian Brooks
Senior server administrator - Sensewhere
Re: Upgrade to 2.4
Posted by Ian Brooks <i....@sensewhere.com>.
Thanks,
Ill give it a try.
-Ian
On Friday 30 May 2014 19:43:00 Zesheng Wu wrote:
> Hi Ian,
> -rollingUpgrade is available since hadoop 2.4, so you can't use
> -rollingUpgrade to upgrade a 2.3 cluster to 2.4.
> Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
> whole cluster and then upgrade the cluster.
> I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
> are some details, wish this can give some help to you:
>
> 1. Put the active NN into safemode: dfsadmin -safemode enter
> 2. Perform a saveNamespace operation: dfsadmin -saveNamespace
> 3. For each component you are using, back up configuration data, databases,
> and other important files
> 4. Shutdown hadoop serverice across your entire cluster
> 5. Check each host to make sure that there are no processes running
> hdfs/yarn
> 6. Back up your HDFS metadata
> 7. Upgrade and start the Journal Nodes and ZKFC
> 8. Make sure that all the journal nodes and zkfc have started normally
> 9. Execute this upgrade operation on active NN: namenode -upgrade
> (Important)
> 10. Wait and make sure from ANN's log that upgrade is completed (from log)
> 11. Bootstrap standby NN: namenode -bootstrapStandby
> 12. Start standby NN: namenode start
> 13. Start each of the DNs: datanode start
> 14. Make sure that everything is running smoothly, this could take a matter
> of days, or even weeks
> 15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
>
>
> 2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
>
> > Hi,
> >
> > I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> > would be useful to us. Unless i'm missing something, I can't see how to
> > upgrade from 2.3 to 2.4
> >
> > -rollingUpgrade doesnt work as the old namenode doesn't understand the
> > command and -upgrade is no longer available.
> >
> > I had a quick google and can't find any documentation on this upgrade,
> > does anyone know how its supposed to be done?
> >
> > -Ian
> >
>
>
>
>
--
-Ian Brooks
Senior server administrator - Sensewhere
Re: Upgrade to 2.4
Posted by Ian Brooks <i....@sensewhere.com>.
Thanks,
Ill give it a try.
-Ian
On Friday 30 May 2014 19:43:00 Zesheng Wu wrote:
> Hi Ian,
> -rollingUpgrade is available since hadoop 2.4, so you can't use
> -rollingUpgrade to upgrade a 2.3 cluster to 2.4.
> Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
> whole cluster and then upgrade the cluster.
> I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
> are some details, wish this can give some help to you:
>
> 1. Put the active NN into safemode: dfsadmin -safemode enter
> 2. Perform a saveNamespace operation: dfsadmin -saveNamespace
> 3. For each component you are using, back up configuration data, databases,
> and other important files
> 4. Shutdown hadoop serverice across your entire cluster
> 5. Check each host to make sure that there are no processes running
> hdfs/yarn
> 6. Back up your HDFS metadata
> 7. Upgrade and start the Journal Nodes and ZKFC
> 8. Make sure that all the journal nodes and zkfc have started normally
> 9. Execute this upgrade operation on active NN: namenode -upgrade
> (Important)
> 10. Wait and make sure from ANN's log that upgrade is completed (from log)
> 11. Bootstrap standby NN: namenode -bootstrapStandby
> 12. Start standby NN: namenode start
> 13. Start each of the DNs: datanode start
> 14. Make sure that everything is running smoothly, this could take a matter
> of days, or even weeks
> 15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
>
>
> 2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
>
> > Hi,
> >
> > I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> > would be useful to us. Unless i'm missing something, I can't see how to
> > upgrade from 2.3 to 2.4
> >
> > -rollingUpgrade doesnt work as the old namenode doesn't understand the
> > command and -upgrade is no longer available.
> >
> > I had a quick google and can't find any documentation on this upgrade,
> > does anyone know how its supposed to be done?
> >
> > -Ian
> >
>
>
>
>
--
-Ian Brooks
Senior server administrator - Sensewhere
Re: Upgrade to 2.4
Posted by Zesheng Wu <wu...@gmail.com>.
Hi Ian,
-rollingUpgrade is available since hadoop 2.4, so you can't use
-rollingUpgrade to upgrade a 2.3 cluster to 2.4.
Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
whole cluster and then upgrade the cluster.
I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
are some details, wish this can give some help to you:
1. Put the active NN into safemode: dfsadmin -safemode enter
2. Perform a saveNamespace operation: dfsadmin -saveNamespace
3. For each component you are using, back up configuration data, databases,
and other important files
4. Shutdown hadoop serverice across your entire cluster
5. Check each host to make sure that there are no processes running
hdfs/yarn
6. Back up your HDFS metadata
7. Upgrade and start the Journal Nodes and ZKFC
8. Make sure that all the journal nodes and zkfc have started normally
9. Execute this upgrade operation on active NN: namenode -upgrade
(Important)
10. Wait and make sure from ANN's log that upgrade is completed (from log)
11. Bootstrap standby NN: namenode -bootstrapStandby
12. Start standby NN: namenode start
13. Start each of the DNs: datanode start
14. Make sure that everything is running smoothly, this could take a matter
of days, or even weeks
15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
> Hi,
>
> I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> would be useful to us. Unless i'm missing something, I can't see how to
> upgrade from 2.3 to 2.4
>
> -rollingUpgrade doesnt work as the old namenode doesn't understand the
> command and -upgrade is no longer available.
>
> I had a quick google and can't find any documentation on this upgrade,
> does anyone know how its supposed to be done?
>
> -Ian
>
--
Best Wishes!
Yours, Zesheng
Re: Upgrade to 2.4
Posted by Zesheng Wu <wu...@gmail.com>.
Hi Ian,
-rollingUpgrade is available since hadoop 2.4, so you can't use
-rollingUpgrade to upgrade a 2.3 cluster to 2.4.
Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
whole cluster and then upgrade the cluster.
I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
are some details, wish this can give some help to you:
1. Put the active NN into safemode: dfsadmin -safemode enter
2. Perform a saveNamespace operation: dfsadmin -saveNamespace
3. For each component you are using, back up configuration data, databases,
and other important files
4. Shutdown hadoop serverice across your entire cluster
5. Check each host to make sure that there are no processes running
hdfs/yarn
6. Back up your HDFS metadata
7. Upgrade and start the Journal Nodes and ZKFC
8. Make sure that all the journal nodes and zkfc have started normally
9. Execute this upgrade operation on active NN: namenode -upgrade
(Important)
10. Wait and make sure from ANN's log that upgrade is completed (from log)
11. Bootstrap standby NN: namenode -bootstrapStandby
12. Start standby NN: namenode start
13. Start each of the DNs: datanode start
14. Make sure that everything is running smoothly, this could take a matter
of days, or even weeks
15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
> Hi,
>
> I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> would be useful to us. Unless i'm missing something, I can't see how to
> upgrade from 2.3 to 2.4
>
> -rollingUpgrade doesnt work as the old namenode doesn't understand the
> command and -upgrade is no longer available.
>
> I had a quick google and can't find any documentation on this upgrade,
> does anyone know how its supposed to be done?
>
> -Ian
>
--
Best Wishes!
Yours, Zesheng
Re: Upgrade to 2.4
Posted by Zesheng Wu <wu...@gmail.com>.
Hi Ian,
-rollingUpgrade is available since hadoop 2.4, so you can't use
-rollingUpgrade to upgrade a 2.3 cluster to 2.4.
Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
whole cluster and then upgrade the cluster.
I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
are some details, wish this can give some help to you:
1. Put the active NN into safemode: dfsadmin -safemode enter
2. Perform a saveNamespace operation: dfsadmin -saveNamespace
3. For each component you are using, back up configuration data, databases,
and other important files
4. Shutdown hadoop serverice across your entire cluster
5. Check each host to make sure that there are no processes running
hdfs/yarn
6. Back up your HDFS metadata
7. Upgrade and start the Journal Nodes and ZKFC
8. Make sure that all the journal nodes and zkfc have started normally
9. Execute this upgrade operation on active NN: namenode -upgrade
(Important)
10. Wait and make sure from ANN's log that upgrade is completed (from log)
11. Bootstrap standby NN: namenode -bootstrapStandby
12. Start standby NN: namenode start
13. Start each of the DNs: datanode start
14. Make sure that everything is running smoothly, this could take a matter
of days, or even weeks
15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
> Hi,
>
> I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> would be useful to us. Unless i'm missing something, I can't see how to
> upgrade from 2.3 to 2.4
>
> -rollingUpgrade doesnt work as the old namenode doesn't understand the
> command and -upgrade is no longer available.
>
> I had a quick google and can't find any documentation on this upgrade,
> does anyone know how its supposed to be done?
>
> -Ian
>
--
Best Wishes!
Yours, Zesheng
Re: Upgrade to 2.4
Posted by Zesheng Wu <wu...@gmail.com>.
Hi Ian,
-rollingUpgrade is available since hadoop 2.4, so you can't use
-rollingUpgrade to upgrade a 2.3 cluster to 2.4.
Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the
whole cluster and then upgrade the cluster.
I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following
are some details, wish this can give some help to you:
1. Put the active NN into safemode: dfsadmin -safemode enter
2. Perform a saveNamespace operation: dfsadmin -saveNamespace
3. For each component you are using, back up configuration data, databases,
and other important files
4. Shutdown hadoop serverice across your entire cluster
5. Check each host to make sure that there are no processes running
hdfs/yarn
6. Back up your HDFS metadata
7. Upgrade and start the Journal Nodes and ZKFC
8. Make sure that all the journal nodes and zkfc have started normally
9. Execute this upgrade operation on active NN: namenode -upgrade
(Important)
10. Wait and make sure from ANN's log that upgrade is completed (from log)
11. Bootstrap standby NN: namenode -bootstrapStandby
12. Start standby NN: namenode start
13. Start each of the DNs: datanode start
14. Make sure that everything is running smoothly, this could take a matter
of days, or even weeks
15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade
2014-05-30 17:54 GMT+08:00 Ian Brooks <i....@sensewhere.com>:
> Hi,
>
> I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades
> would be useful to us. Unless i'm missing something, I can't see how to
> upgrade from 2.3 to 2.4
>
> -rollingUpgrade doesnt work as the old namenode doesn't understand the
> command and -upgrade is no longer available.
>
> I had a quick google and can't find any documentation on this upgrade,
> does anyone know how its supposed to be done?
>
> -Ian
>
--
Best Wishes!
Yours, Zesheng