You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/06/23 04:50:51 UTC
Review Request 35764: Installing Repo Packages needs to be more
robust to
handle the actual_version installed when the script is killed because of a
timeout
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/
-----------------------------------------------------------
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Repository: ambari
Description
-------
When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
Diffs
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
Diff: https://reviews.apache.org/r/35764/diff/
Testing
-------
Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
Repo version including build number:
* Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
* Failure after installing the first package, retries eventually passed.
Repo version without build numbner:
* Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
* Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
* Eventually, it passed.
Still need to fix unit tests.
Thanks,
Alejandro Fernandez
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/
-----------------------------------------------------------
(Updated June 23, 2015, 6:20 p.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Changes
-------
Changed name of file to *.json
Bugs: AMBARI-12083
https://issues.apache.org/jira/browse/AMBARI-12083
Repository: ambari
Description
-------
When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
Diff: https://reviews.apache.org/r/35764/diff/
Testing
-------
Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
Repo version including build number:
* Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
* Failure after installing the first package, retries eventually passed.
Repo version without build numbner:
* Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
* Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
* Eventually, it passed.
----------------------------------------------------------------------
Total run:762
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/#review88945
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (line 128)
<https://reviews.apache.org/r/35764/#comment141552>
different logging calls
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (line 140)
<https://reviews.apache.org/r/35764/#comment141553>
different logging calls
- Dmitro Lisnichenko
On June 23, 2015, 3:04 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35764/
> -----------------------------------------------------------
>
> (Updated June 23, 2015, 3:04 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12083
> https://issues.apache.org/jira/browse/AMBARI-12083
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
> The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
>
> For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
>
> Diff: https://reviews.apache.org/r/35764/diff/
>
>
> Testing
> -------
>
> Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
>
> Repo version including build number:
> * Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
> * Failure after installing the first package, retries eventually passed.
>
> Repo version without build numbner:
> * Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
> * Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
> * Eventually, it passed.
>
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/#review88999
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (lines 118 - 123)
<https://reviews.apache.org/r/35764/#comment141596>
2.3.0.0-1 already includes the build number. Are we supposed to correct that too?
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (line 128)
<https://reviews.apache.org/r/35764/#comment141592>
Logger.logger.exception will also include the stack trace.
- Alejandro Fernandez
On June 23, 2015, 3:04 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35764/
> -----------------------------------------------------------
>
> (Updated June 23, 2015, 3:04 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12083
> https://issues.apache.org/jira/browse/AMBARI-12083
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
> The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
>
> For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
>
> Diff: https://reviews.apache.org/r/35764/diff/
>
>
> Testing
> -------
>
> Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
>
> Repo version including build number:
> * Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
> * Failure after installing the first package, retries eventually passed.
>
> Repo version without build numbner:
> * Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
> * Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
> * Eventually, it passed.
>
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/#review88949
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (line 53)
<https://reviews.apache.org/r/35764/#comment141554>
Should call it .json; that's consistent with the content in the file.
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (lines 118 - 123)
<https://reviews.apache.org/r/35764/#comment141555>
This will be incorrect if the user ends up putting, say 2.3.0.0-1 and the real determined is 2.3.0.0-1234.
- Nate Cole
On June 22, 2015, 11:04 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35764/
> -----------------------------------------------------------
>
> (Updated June 22, 2015, 11:04 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12083
> https://issues.apache.org/jira/browse/AMBARI-12083
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
> The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
>
> For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
>
> Diff: https://reviews.apache.org/r/35764/diff/
>
>
> Testing
> -------
>
> Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
>
> Repo version including build number:
> * Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
> * Failure after installing the first package, retries eventually passed.
>
> Repo version without build numbner:
> * Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
> * Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
> * Eventually, it passed.
>
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/#review89001
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On June 22, 2015, 11:04 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35764/
> -----------------------------------------------------------
>
> (Updated June 22, 2015, 11:04 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12083
> https://issues.apache.org/jira/browse/AMBARI-12083
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
> The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
>
> For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
>
> Diff: https://reviews.apache.org/r/35764/diff/
>
>
> Testing
> -------
>
> Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
>
> Repo version including build number:
> * Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
> * Failure after installing the first package, retries eventually passed.
>
> Repo version without build numbner:
> * Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
> * Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
> * Eventually, it passed.
>
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/
-----------------------------------------------------------
(Updated June 23, 2015, 3:04 a.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Changes
-------
Fixed unit tests.
Bugs: AMBARI-12083
https://issues.apache.org/jira/browse/AMBARI-12083
Repository: ambari
Description
-------
When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
ambari-server/src/test/python/custom_actions/TestInstallPackages.py 763f9e2
Diff: https://reviews.apache.org/r/35764/diff/
Testing (updated)
-------
Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
Repo version including build number:
* Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
* Failure after installing the first package, retries eventually passed.
Repo version without build numbner:
* Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
* Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
* Eventually, it passed.
----------------------------------------------------------------------
Total run:762
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/#review88908
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py (line 55)
<https://reviews.apache.org/r/35764/#comment141526>
Unrelated bug that I needed to fix.
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (line 164)
<https://reviews.apache.org/r/35764/#comment141528>
I decided to use the file because there is no guarantee that the structured out will eventually make it to the request, and the server will accept it.
Robustness!
- Alejandro Fernandez
On June 23, 2015, 2:51 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35764/
> -----------------------------------------------------------
>
> (Updated June 23, 2015, 2:51 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12083
> https://issues.apache.org/jira/browse/AMBARI-12083
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
> The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
>
> For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
>
> Diff: https://reviews.apache.org/r/35764/diff/
>
>
> Testing
> -------
>
> Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
>
> Repo version including build number:
> * Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
> * Failure after installing the first package, retries eventually passed.
>
> Repo version without build numbner:
> * Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
> * Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
> * Eventually, it passed.
>
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 35764: Installing Repo Packages needs to be more
robust
to handle the actual_version installed when the script is killed because of
a timeout
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35764/
-----------------------------------------------------------
(Updated June 23, 2015, 2:51 a.m.)
Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
Bugs: AMBARI-12083
https://issues.apache.org/jira/browse/AMBARI-12083
Repository: ambari
Description
-------
When installing bits of a new repo to perform an RU, if the repo version does not contain a build number, then the actual_version has to be calculated.
The problem is that this information can only be retrieved by calculating the delta of the versions when calling "hdp-select versions", so the only the first time will return a value. If the script is killed, then the actual_version will never be calculated and returned to the server.
For this reason, the package installer must tolerate receiving a SIGINT or SIGTERM, and calculate the actual_version. Further, it helps to store it in a file, in case that the response or something upstream results in another failure.
Diffs
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py bb3faaa
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 5f7ac90
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py ffe9815
Diff: https://reviews.apache.org/r/35764/diff/
Testing
-------
Verified that the following works by decreasing the install package timeout to 150 secs, and trying the following combinations
Repo version including build number:
* Failure during installing the first package. Because actual_version was already set, did not need to call "hdp-select versions".
* Failure after installing the first package, retries eventually passed.
Repo version without build numbner:
* Failure during installing the first package. Because actual_version was none, it tried calling "hdp-select versions" but the newer version was not yet installed, so was allowed to retry.
* Failure after installing the first package. Beacuse "hdp-select versions" reported a value in the delta, it was saved to the text file. Since it timed out, was allowed to retry.
* Eventually, it passed.
Still need to fix unit tests.
Thanks,
Alejandro Fernandez