You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/11/04 22:52:07 UTC
Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3 must
uninstall 2.1 bits before starting on 2.3
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
Bugs: AMBARI-13723
https://issues.apache.org/jira/browse/AMBARI-13723
Repository: ambari
Description
-------
We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
**Manual Upgrade Instructions:**
* Stop all services using 2.1
* yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
* yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, and hence MR Service Check passes with "hadoop" command
**Express Upgrade:**
* Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
* Stop all services using 2.1
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
Diffs
-----
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
Diff: https://reviews.apache.org/r/39955/diff/
Testing
-------
Verified during Express Upgrade from HDP 2.1->2.3
Thanks,
Alejandro Fernandez
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/#review105164
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml (line 190)
<https://reviews.apache.org/r/39955/#comment163582>
Just "Remove HDP-2.1" no need to say "bits" :)
ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml (line 196)
<https://reviews.apache.org/r/39955/#comment163584>
"will delete HDP-2.1." "This is because 2.1 did not..."
- Nate Cole
On Nov. 4, 2015, 5:38 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39955/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2015, 5:38 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-13723
> https://issues.apache.org/jira/browse/AMBARI-13723
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
>
> **Manual Upgrade Instructions:**
>
> * Stop all services using 2.1
> * yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
> * yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, and hence MR Service Check passes with "hadoop" command
>
> **Express Upgrade:**
>
> * Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
> * Stop all services using 2.1
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
>
> One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
> Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
>
> Diff: https://reviews.apache.org/r/39955/diff/
>
>
> Testing
> -------
>
> Verified during Express Upgrade from HDP 2.1->2.3
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Nov. 4, 2015, 11 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py, line 46
> > <https://reviews.apache.org/r/39955/diff/2/?file=1116020#file1116020line46>
> >
> > Instead of hardcoding the list of packages, cant we build the list of packages similar to how we do for Install Packages from the stack definition? That way it would work for all OSes.
We don't have a very good way of doing this right now, since it would involve a custom command to pass in configs.
For the time being, this works. Closing this issue.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/#review105168
-----------------------------------------------------------
On Nov. 4, 2015, 11:25 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39955/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2015, 11:25 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-13723
> https://issues.apache.org/jira/browse/AMBARI-13723
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
>
> **Manual Upgrade Instructions:**
>
> * Stop all services using 2.1
> * yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
> * yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, and hence MR Service Check passes with "hadoop" command
>
> **Express Upgrade:**
>
> * Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
> * Stop all services using 2.1
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
>
> One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
> Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
>
> Diff: https://reviews.apache.org/r/39955/diff/
>
>
> Testing
> -------
>
> Verified during Express Upgrade from HDP 2.1->2.3
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/#review105168
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py (line 46)
<https://reviews.apache.org/r/39955/#comment163588>
Instead of hardcoding the list of packages, cant we build the list of packages similar to how we do for Install Packages from the stack definition? That way it would work for all OSes.
- Jayush Luniya
On Nov. 4, 2015, 10:38 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39955/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2015, 10:38 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-13723
> https://issues.apache.org/jira/browse/AMBARI-13723
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
>
> **Manual Upgrade Instructions:**
>
> * Stop all services using 2.1
> * yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
> * yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, and hence MR Service Check passes with "hadoop" command
>
> **Express Upgrade:**
>
> * Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
> * Stop all services using 2.1
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
>
> One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
> Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
>
> Diff: https://reviews.apache.org/r/39955/diff/
>
>
> Testing
> -------
>
> Verified during Express Upgrade from HDP 2.1->2.3
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/#review105223
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 5, 2015, 1:25 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39955/
> -----------------------------------------------------------
>
> (Updated Nov. 5, 2015, 1:25 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-13723
> https://issues.apache.org/jira/browse/AMBARI-13723
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
>
> **Manual Upgrade Instructions:**
>
> * Stop all services using 2.1
> * yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
> * yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, and hence MR Service Check passes with "hadoop" command
>
> **Express Upgrade:**
>
> * Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
> * Stop all services using 2.1
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
>
> One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
> Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
>
> Diff: https://reviews.apache.org/r/39955/diff/
>
>
> Testing
> -------
>
> Verified during Express Upgrade from HDP 2.1->2.3
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/
-----------------------------------------------------------
(Updated Nov. 4, 2015, 11:25 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
Changes
-------
Changed text message.
Bugs: AMBARI-13723
https://issues.apache.org/jira/browse/AMBARI-13723
Repository: ambari
Description
-------
We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
**Manual Upgrade Instructions:**
* Stop all services using 2.1
* yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
* yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, and hence MR Service Check passes with "hadoop" command
**Express Upgrade:**
* Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
* Stop all services using 2.1
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
Diffs (updated)
-----
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
Diff: https://reviews.apache.org/r/39955/diff/
Testing
-------
Verified during Express Upgrade from HDP 2.1->2.3
Thanks,
Alejandro Fernandez
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/#review105162
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py (line 46)
<https://reviews.apache.org/r/39955/#comment163580>
We may need to test this on other OS'es.
- Alejandro Fernandez
On Nov. 4, 2015, 10:38 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39955/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2015, 10:38 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-13723
> https://issues.apache.org/jira/browse/AMBARI-13723
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
>
> **Manual Upgrade Instructions:**
>
> * Stop all services using 2.1
> * yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
> * yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, and hence MR Service Check passes with "hadoop" command
>
> **Express Upgrade:**
>
> * Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
> * Stop all services using 2.1
> * Change the version to 2.3 (hdp-select, conf-select, etc.)
> * Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
>
> One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
> Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
>
> Diff: https://reviews.apache.org/r/39955/diff/
>
>
> Testing
> -------
>
> Verified during Express Upgrade from HDP 2.1->2.3
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 39955: AMBARI-13723. Express Upgrade: HDP 2.1->2.3
must uninstall 2.1 bits before starting on 2.3
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39955/
-----------------------------------------------------------
(Updated Nov. 4, 2015, 10:38 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jayush Luniya, Nate Cole, and Swapan Shridhar.
Bugs: AMBARI-13723
https://issues.apache.org/jira/browse/AMBARI-13723
Repository: ambari
Description
-------
We found a major flaw in the Express Upgrade from 2.1->2.3. Basically, "hadoop" bin still points to the old binary because we had to install HDP 2.3 while 2.1 bits still existed.
**Manual Upgrade Instructions:**
* Stop all services using 2.1
* yum erase hadoop hdfs hive … (this will remove scripts /usr/bin/hadoop, /usr/bin/hdfs, /usr/bin/hive, etc/)
* yum install hadoop_2_3_* hdfs_2_3_* hive_2_3_* (rpm should point /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop)
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, and hence MR Service Check passes with "hadoop" command
**Express Upgrade:**
* Install HDP 2.3 bits. We reused the RU framework that relies on a 2 step process: install bits, perform upgrade. Hence, /usr/bin/hadoop is still using 2.1 bits
* Stop all services using 2.1
* Change the version to 2.3 (hdp-select, conf-select, etc.)
* Start all services, will fail on MR Service Check because ExecuteHadoop() resource calls "hadoop" command, which is still pointing to 2.1 bits!
One solution is for the orchestration to remove the 2.1 bits before starting using 2.3.
Uninstalling the 2.1 bits will remove /usr/bin/hadoop, but when we call "hdp-select set hadoop-client <new_version>" it will create the symlink /usr/bin/hadoop -> /usr/hdp/current/hadoop-client/bin/hadoop
Diffs (updated)
-----
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 91d4f64
Diff: https://reviews.apache.org/r/39955/diff/
Testing
-------
Verified during Express Upgrade from HDP 2.1->2.3
Thanks,
Alejandro Fernandez