You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2015/11/09 14:20:17 UTC
Review Request 40080: AMBARI-13748: Being able to clean repository
cache before installing open source components
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/
-----------------------------------------------------------
Review request for Ambari and Alejandro Fernandez.
Bugs: AMBARI-13748
https://issues.apache.org/jira/browse/AMBARI-13748
Repository: ambari
Description
-------
Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
Diff: https://reviews.apache.org/r/40080/diff/
Testing
-------
New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
Thanks,
Di Li
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106068
-----------------------------------------------------------
Also one more problem I see is that some users request to handle install tasks when apt-get or other are under lock and it's not possible to use it. Since they have 'apt-get check' run on cron and don't want the tasks to fail. I'm going to implement retrties for such a commands if lock file exists. I think here simmilar problem appears.
- Andrew Onischuk
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 4:03 p.m., Andrew Onischuk wrote:
> > Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
>
> Di Li wrote:
> hello Andrew,
>
> Thanks for the explanation. I will just use this one for all your comments
>
> 1. Repo vs package install in the basic case: I guess one way to deal with this is to create repo on each agent before installing packages. This require some serious changes to the way agent install packages, I suppose. I get it that trying to create the repo file per package install can make sure the repo file content is always up to date.
>
> 2. Locks: it's not being handled right now.
>
> 3. Purpose of the jira: This is a very special case where the cache contains out of date info of HDP which may cause the install pacakge steps to fail when user tries to run rolling upgrade or do a fresh install. This may happen for a cluster previous installed with HDP (when user does a floppy job on clean up the cluster or clean up the repo cache properly). The logic here is try to do some cache cleaning when Ambari creates the repo file on each Agent node.
>
> Just as you said, new repo logic in SLES/RHEL is perfectly fine, for a cluster that has never been installed with HDP before or when user properly cleans up repo cache.
>
> Andrew Onischuk wrote:
> Di, ok I get it but the problem is slowness. My proposal to fix this would be the way we have done this for apt-get intall vs apt-get update.
> (check apt.py in re
>
> Please check apt.py.
> if code:
> Logger.info("Execution of '%s' returned %d. %s" % (cmd, code, out))
> Logger.info("Failed to install package %s. Executing `%s`" % (name, string_cmd_from_args_list(REPO_UPDATE_CMD)))
> code, out = shell.call(REPO_UPDATE_CMD, sudo=True, logoutput=self.get_logoutput())
>
> My proposal to fix this in the same way. meaning:
>
> 1. Try to install package
> 2. If it fails we do 'yum clean all'
> 3. Try again, if it fails, fail the task.
>
>
> I think this is the way this issue should be handled
>
> regards,
> Andrew.
>
> Andrew Onischuk wrote:
> Also please don't forget to add log messages about what is done at every point. Like in the example I gave you above.
Di Li can I get your skype? Or can you add me empower-666. To get this conversation faster on?
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 4:03 p.m., Andrew Onischuk wrote:
> > Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
>
> Di Li wrote:
> hello Andrew,
>
> Thanks for the explanation. I will just use this one for all your comments
>
> 1. Repo vs package install in the basic case: I guess one way to deal with this is to create repo on each agent before installing packages. This require some serious changes to the way agent install packages, I suppose. I get it that trying to create the repo file per package install can make sure the repo file content is always up to date.
>
> 2. Locks: it's not being handled right now.
>
> 3. Purpose of the jira: This is a very special case where the cache contains out of date info of HDP which may cause the install pacakge steps to fail when user tries to run rolling upgrade or do a fresh install. This may happen for a cluster previous installed with HDP (when user does a floppy job on clean up the cluster or clean up the repo cache properly). The logic here is try to do some cache cleaning when Ambari creates the repo file on each Agent node.
>
> Just as you said, new repo logic in SLES/RHEL is perfectly fine, for a cluster that has never been installed with HDP before or when user properly cleans up repo cache.
>
> Andrew Onischuk wrote:
> Di, ok I get it but the problem is slowness. My proposal to fix this would be the way we have done this for apt-get intall vs apt-get update.
> (check apt.py in re
>
> Please check apt.py.
> if code:
> Logger.info("Execution of '%s' returned %d. %s" % (cmd, code, out))
> Logger.info("Failed to install package %s. Executing `%s`" % (name, string_cmd_from_args_list(REPO_UPDATE_CMD)))
> code, out = shell.call(REPO_UPDATE_CMD, sudo=True, logoutput=self.get_logoutput())
>
> My proposal to fix this in the same way. meaning:
>
> 1. Try to install package
> 2. If it fails we do 'yum clean all'
> 3. Try again, if it fails, fail the task.
>
>
> I think this is the way this issue should be handled
>
> regards,
> Andrew.
Also please don't forget to add log messages about what is done at every point. Like in the example I gave you above.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 4:03 p.m., Andrew Onischuk wrote:
> > Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
hello Andrew,
Thanks for the explanation. I will just use this one for all your comments
1. Repo vs package install in the basic case: I guess one way to deal with this is to create repo on each agent before installing packages. This require some serious changes to the way agent install packages, I suppose. I get it that trying to create the repo file per package install can make sure the repo file content is always up to date.
2. Locks: it's not being handled right now.
3. Purpose of the jira: This is a very special case where the cache contains out of date info of HDP which may cause the install pacakge steps to fail when user tries to run rolling upgrade or do a fresh install. This may happen for a cluster previous installed with HDP (when user does a floppy job on clean up the cluster or clean up the repo cache properly). The logic here is try to do some cache cleaning when Ambari creates the repo file on each Agent node.
Just as you said, new repo logic in SLES/RHEL is perfectly fine, for a cluster that has never been installed with HDP before or when user properly cleans up repo cache.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 4:03 p.m., Andrew Onischuk wrote:
> > Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
>
> Di Li wrote:
> hello Andrew,
>
> Thanks for the explanation. I will just use this one for all your comments
>
> 1. Repo vs package install in the basic case: I guess one way to deal with this is to create repo on each agent before installing packages. This require some serious changes to the way agent install packages, I suppose. I get it that trying to create the repo file per package install can make sure the repo file content is always up to date.
>
> 2. Locks: it's not being handled right now.
>
> 3. Purpose of the jira: This is a very special case where the cache contains out of date info of HDP which may cause the install pacakge steps to fail when user tries to run rolling upgrade or do a fresh install. This may happen for a cluster previous installed with HDP (when user does a floppy job on clean up the cluster or clean up the repo cache properly). The logic here is try to do some cache cleaning when Ambari creates the repo file on each Agent node.
>
> Just as you said, new repo logic in SLES/RHEL is perfectly fine, for a cluster that has never been installed with HDP before or when user properly cleans up repo cache.
Di, ok I get it but the problem is slowness. My proposal to fix this would be the way we have done this for apt-get intall vs apt-get update.
(check apt.py in re
Please check apt.py.
if code:
Logger.info("Execution of '%s' returned %d. %s" % (cmd, code, out))
Logger.info("Failed to install package %s. Executing `%s`" % (name, string_cmd_from_args_list(REPO_UPDATE_CMD)))
code, out = shell.call(REPO_UPDATE_CMD, sudo=True, logoutput=self.get_logoutput())
My proposal to fix this in the same way. meaning:
1. Try to install package
2. If it fails we do 'yum clean all'
3. Try again, if it fails, fail the task.
I think this is the way this issue should be handled
regards,
Andrew.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 4:03 p.m., Andrew Onischuk wrote:
> > Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
>
> Di Li wrote:
> hello Andrew,
>
> Thanks for the explanation. I will just use this one for all your comments
>
> 1. Repo vs package install in the basic case: I guess one way to deal with this is to create repo on each agent before installing packages. This require some serious changes to the way agent install packages, I suppose. I get it that trying to create the repo file per package install can make sure the repo file content is always up to date.
>
> 2. Locks: it's not being handled right now.
>
> 3. Purpose of the jira: This is a very special case where the cache contains out of date info of HDP which may cause the install pacakge steps to fail when user tries to run rolling upgrade or do a fresh install. This may happen for a cluster previous installed with HDP (when user does a floppy job on clean up the cluster or clean up the repo cache properly). The logic here is try to do some cache cleaning when Ambari creates the repo file on each Agent node.
>
> Just as you said, new repo logic in SLES/RHEL is perfectly fine, for a cluster that has never been installed with HDP before or when user properly cleans up repo cache.
>
> Andrew Onischuk wrote:
> Di, ok I get it but the problem is slowness. My proposal to fix this would be the way we have done this for apt-get intall vs apt-get update.
> (check apt.py in re
>
> Please check apt.py.
> if code:
> Logger.info("Execution of '%s' returned %d. %s" % (cmd, code, out))
> Logger.info("Failed to install package %s. Executing `%s`" % (name, string_cmd_from_args_list(REPO_UPDATE_CMD)))
> code, out = shell.call(REPO_UPDATE_CMD, sudo=True, logoutput=self.get_logoutput())
>
> My proposal to fix this in the same way. meaning:
>
> 1. Try to install package
> 2. If it fails we do 'yum clean all'
> 3. Try again, if it fails, fail the task.
>
>
> I think this is the way this issue should be handled
>
> regards,
> Andrew.
>
> Andrew Onischuk wrote:
> Also please don't forget to add log messages about what is done at every point. Like in the example I gave you above.
>
> Andrew Onischuk wrote:
> Di Li can I get your skype? Or can you add me empower-666. To get this conversation faster on?
hello Andrew,
thanks for the pointers, I will work on your proposed approach and submit a new patch as soon as possible.
Thank you.
Di
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106074
-----------------------------------------------------------
Di, also one more question. As far as I know, new repositories are automatically added to cache on suse/rhel when first yum/zypper install is triggered. Do we target something else here?
- Andrew Onischuk
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 3:39 p.m., Andrew Onischuk wrote:
> > -1 for such an approach this will make repo data cleaned on every install task. Which will make package installation in every task refresh all repositories which makes all the installs much slower.
> > If we really want to do this we should do it once, and not every install task.
This is the reason why it wasn't implemented in the first place. Because there was a bunch of issues like this from guys testing Ambari. We should think of better mechanism to handle this issue.
P.s. Please see https://issues.apache.org/jira/browse/AMBARI-10992 some customers are really critical about the tasks times.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106066
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 3:39 p.m., Andrew Onischuk wrote:
> > -1 for such an approach this will make repo data cleaned on every install task. Which will make package installation in every task refresh all repositories which makes all the installs much slower.
> > If we really want to do this we should do it once, and not every install task.
>
> Andrew Onischuk wrote:
> This is the reason why it wasn't implemented in the first place. Because there was a bunch of issues like this from guys testing Ambari. We should think of better mechanism to handle this issue.
>
> P.s. Please see https://issues.apache.org/jira/browse/AMBARI-10992 some customers are really critical about the tasks times.
hello Andrew,
Clean cache logic is in the Repository Python scripts, it happens after the repo file is installed, not per package install.
For rolling upgrade, after the repo file is created on each node, the clean cache logic runs against the specific repo ids used for the rolling upgrade, and that is it, all subsequent package installations will not trigger the cache clean logic.
For fresh install,
also after the repo file is created on each node, the same clean cache logic runs. The repo init logic in the BEFORE-INSTALL hook repo_initialization.py script also has a check to skip the whole repo creation (which includes the clean cache) based on "host_sys_prepped" property value from params.py.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106066
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 3:39 p.m., Andrew Onischuk wrote:
> > -1 for such an approach this will make repo data cleaned on every install task. Which will make package installation in every task refresh all repositories which makes all the installs much slower.
> > If we really want to do this we should do it once, and not every install task.
>
> Andrew Onischuk wrote:
> This is the reason why it wasn't implemented in the first place. Because there was a bunch of issues like this from guys testing Ambari. We should think of better mechanism to handle this issue.
>
> P.s. Please see https://issues.apache.org/jira/browse/AMBARI-10992 some customers are really critical about the tasks times.
>
> Di Li wrote:
> hello Andrew,
>
> Clean cache logic is in the Repository Python scripts, it happens after the repo file is installed, not per package install.
>
> For rolling upgrade, after the repo file is created on each node, the clean cache logic runs against the specific repo ids used for the rolling upgrade, and that is it, all subsequent package installations will not trigger the cache clean logic.
>
> For fresh install,
>
> also after the repo file is created on each node, the same clean cache logic runs. The repo init logic in the BEFORE-INSTALL hook repo_initialization.py script also has a check to skip the whole repo creation (which includes the clean cache) based on "host_sys_prepped" property value from params.py.
Repository install occurs on every INSTALL task in basic case.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106066
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106066
-----------------------------------------------------------
-1 for such an approach this will make repo data cleaned on every install task. Which will make package installation in every task refresh all repositories which makes all the installs much slower.
If we really want to do this we should do it once, and not every install task.
- Andrew Onischuk
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review105955
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
>
> Di Li wrote:
> hello folks,
>
> Just to make sure I understand it:
> 1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
> 2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
>
> Thanks.
>
> Andrew Onischuk wrote:
> 1. Yes
> 2. Yes, it's a separate python file, and Alejandro probably mean "custom_action" not custom_command. You can check install_packages.py and other scripts in that folder for an example.
>
> Jayush Luniya wrote:
> +1 on custom_action route.
>
> Di Li wrote:
> hello folks,
>
> There is already this update_repo.py custom_action Python script in the current custom_actions directory. The python script simply creates HDP repo files. Should it be piggy-backed to run yum clean all right after the repos are created, instead of having a desinaged clean repo cache custom_action ?
>
> Thanks.
>
> Di
>
> Alejandro Fernandez wrote:
> My preference is a custom action to clean repos on-demand, since this is very explicit and clear.
Hello Alejandro and Andrew,
could you please help review the custom action code change I just made.
Thanks.
Di
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
>
> Di Li wrote:
> hello folks,
>
> Just to make sure I understand it:
> 1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
> 2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
>
> Thanks.
>
> Andrew Onischuk wrote:
> 1. Yes
> 2. Yes, it's a separate python file, and Alejandro probably mean "custom_action" not custom_command. You can check install_packages.py and other scripts in that folder for an example.
>
> Jayush Luniya wrote:
> +1 on custom_action route.
hello folks,
There is already this update_repo.py custom_action Python script in the current custom_actions directory. The python script simply creates HDP repo files. Should it be piggy-backed to run yum clean all right after the repos are created, instead of having a desinaged clean repo cache custom_action ?
Thanks.
Di
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
Actually I think custom command is a nicer idea, than doing it without customer knowledge.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
>
> Di Li wrote:
> hello folks,
>
> Just to make sure I understand it:
> 1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
> 2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
>
> Thanks.
>
> Andrew Onischuk wrote:
> 1. Yes
> 2. Yes, it's a separate python file, and Alejandro probably mean "custom_action" not custom_command. You can check install_packages.py and other scripts in that folder for an example.
>
> Jayush Luniya wrote:
> +1 on custom_action route.
>
> Di Li wrote:
> hello folks,
>
> There is already this update_repo.py custom_action Python script in the current custom_actions directory. The python script simply creates HDP repo files. Should it be piggy-backed to run yum clean all right after the repos are created, instead of having a desinaged clean repo cache custom_action ?
>
> Thanks.
>
> Di
My preference is a custom action to clean repos on-demand, since this is very explicit and clear.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
hello folks,
Just to make sure I understand it:
1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
Thanks.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
+1 for that. Custom command that can be ran on-demand to clean all repos.
We need to be careful if the yum lock is already acquired, we don't want to wait forever.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
>
> Di Li wrote:
> hello folks,
>
> Just to make sure I understand it:
> 1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
> 2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
>
> Thanks.
1. Yes
2. Yes, it's a separate python file, and Alejandro probably mean "custom_action" not custom_command. You can check install_packages.py and other scripts in that folder for an example.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On Nov. 11, 2015, 6:01 p.m., Andrew Onischuk wrote:
> > Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
>
> Alejandro Fernandez wrote:
> +1 for that. Custom command that can be ran on-demand to clean all repos.
> We need to be careful if the yum lock is already acquired, we don't want to wait forever.
>
> Andrew Onischuk wrote:
> Actually I think custom command is a nicer idea, than doing it without customer knowledge.
>
> Di Li wrote:
> hello folks,
>
> Just to make sure I understand it:
> 1. "clean for all repositories" >> As in simply running "yum clean all" without the --disablerepo and --enablerepo parameters at all?
> 2. "custom command" >> As in a separate Python script that users can run from the command line? Or are we talking about a UI feature here?
>
> Thanks.
>
> Andrew Onischuk wrote:
> 1. Yes
> 2. Yes, it's a separate python file, and Alejandro probably mean "custom_action" not custom_command. You can check install_packages.py and other scripts in that folder for an example.
+1 on custom_action route.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106100
-----------------------------------------------------------
Also can we do the clean for all repositories? That would simply the logics exponentiantialy and help to avoid some bugs.
- Andrew Onischuk
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106098
-----------------------------------------------------------
What about a custom command to clean packages?
- Alejandro Fernandez
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 13, 2015, 9:19 a.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py, line 52
> > <https://reviews.apache.org/r/40080/diff/3/?file=1124429#file1124429line52>
> >
> > I think 'apt-get clean' on Ubuntu cleans all the locally save deb files and etc.
> > While 'apt-get update' updates the repositories information.
> >
> > I think you meant apt-get update here? No?
hello Andrew,
Could you please review the new patch? I updated the apt-get command from against clean to update.
Thanks.
Di
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106386
-----------------------------------------------------------
On Nov. 12, 2015, 9:44 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2015, 9:44 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106386
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py (line 52)
<https://reviews.apache.org/r/40080/#comment165130>
I think 'apt-get clean' on Ubuntu cleans all the locally save deb files and etc.
While 'apt-get update' updates the repositories information.
I think you meant apt-get update here? No?
- Andrew Onischuk
On Nov. 12, 2015, 9:44 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2015, 9:44 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106411
-----------------------------------------------------------
Ship it!
Ship It!
- Andrew Onischuk
On Nov. 13, 2015, 1:56 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2015, 1:56 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
> On Nov. 13, 2015, 6:40 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py, line 22
> > <https://reviews.apache.org/r/40080/diff/4/?file=1125081#file1125081line22>
> >
> > Let's avoid import * since it's been known to cause issues.
hello Alejandro,
I just updated the python script to have explicit imports.
Thanks.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106477
-----------------------------------------------------------
On Nov. 13, 2015, 1:56 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2015, 1:56 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106477
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py (line 22)
<https://reviews.apache.org/r/40080/#comment165191>
Let's avoid import * since it's been known to cause issues.
- Alejandro Fernandez
On Nov. 13, 2015, 1:56 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2015, 1:56 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106742
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Nov. 13, 2015, 7:50 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2015, 7:50 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
> ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
> ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
> New unit test for custom action code change. Also test the custom action by REST API call
>
>
> curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/
-----------------------------------------------------------
(Updated Nov. 13, 2015, 7:50 p.m.)
Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
Bugs: AMBARI-13748
https://issues.apache.org/jira/browse/AMBARI-13748
Repository: ambari
Description
-------
Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
Diffs (updated)
-----
ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
Diff: https://reviews.apache.org/r/40080/diff/
Testing
-------
New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
New unit test for custom action code change. Also test the custom action by REST API call
curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
Thanks,
Di Li
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/
-----------------------------------------------------------
(Updated Nov. 13, 2015, 1:56 p.m.)
Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
Bugs: AMBARI-13748
https://issues.apache.org/jira/browse/AMBARI-13748
Repository: ambari
Description
-------
Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
Diffs (updated)
-----
ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
Diff: https://reviews.apache.org/r/40080/diff/
Testing
-------
New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
New unit test for custom action code change. Also test the custom action by REST API call
curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
Thanks,
Di Li
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/
-----------------------------------------------------------
(Updated Nov. 12, 2015, 9:44 p.m.)
Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
Bugs: AMBARI-13748
https://issues.apache.org/jira/browse/AMBARI-13748
Repository: ambari
Description
-------
Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
Diffs (updated)
-----
ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml e725d68
ambari-server/src/main/resources/custom_actions/scripts/clear_repocache.py PRE-CREATION
ambari-server/src/test/python/custom_actions/TestClearRepoCache.py PRE-CREATION
Diff: https://reviews.apache.org/r/40080/diff/
Testing (updated)
-------
New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
New unit test for custom action code change. Also test the custom action by REST API call
curl -u admin:admin -H "X-Requested-By: admin" -X POST -d'{"RequestInfo":{"context":"clear repo cache test", "action" : "clear_repocache"}, "Requests/resource_filters":[{"hosts":"agent.mydomain.com"}]}' http://server.mydomain.com:8080/api/v1/clusters/bi/requests
Thanks,
Di Li
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review106069
-----------------------------------------------------------
Also one more problem I see is that some users request to handle install tasks when apt-get or other are under lock and it's not possible to use it. Since they have 'apt-get check' run on cron and don't want the tasks to fail. I'm going to implement retrties for such a commands if lock file exists. I think here simmilar problem appears.
- Andrew Onischuk
On Nov. 10, 2015, 9:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/
-----------------------------------------------------------
(Updated Nov. 10, 2015, 9:02 p.m.)
Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jayush Luniya, and Sid Wagle.
Bugs: AMBARI-13748
https://issues.apache.org/jira/browse/AMBARI-13748
Repository: ambari
Description
-------
Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
Diff: https://reviews.apache.org/r/40080/diff/
Testing
-------
New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
Thanks,
Di Li
Re: Review Request 40080: AMBARI-13748: Being able to clean
repository cache before installing open source components
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40080/#review105920
-----------------------------------------------------------
Thanks for the contribution. Please include Andrew Onischuk, Sid Waggle, and Jayush Luniya in the code review.
ambari-common/src/main/python/resource_management/libraries/providers/repository.py (line 97)
<https://reviews.apache.org/r/40080/#comment164611>
Small typo here.
ambari-common/src/main/python/resource_management/libraries/providers/repository.py (line 103)
<https://reviews.apache.org/r/40080/#comment164612>
Should check that output is not None and len > 2
For line_list, also check the len before attempting to get an index.
Do case insensitive comparison to "yes" by casting
to lower(), and same with "suse-" and "opensuse"
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py (line 56)
<https://reviews.apache.org/r/40080/#comment164615>
Should this be a Logger.info?
- Alejandro Fernandez
On Nov. 9, 2015, 1:20 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40080/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2015, 1:20 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-13748
> https://issues.apache.org/jira/browse/AMBARI-13748
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Need a way for Ambari to trigger a repository clean (yum clean expire-cache on rhel, for example) when updating repository files during an Ambari update as well as fresh installation of a cluster.
>
> Currently, this must be done occasionally through a manual step, and if not done will lead to problems during install.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/providers/repository.py 11002cc
> ambari-common/src/main/python/resource_management/libraries/resources/repository.py 2484d67
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 1dfaa54
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 05751fa
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 83b6bb5
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py f9d69a8
>
> Diff: https://reviews.apache.org/r/40080/diff/
>
>
> Testing
> -------
>
> New unit test. Patch cluster with the changes and run fresh install and rolling upgrade to verify the repo clean up logic was run.
>
>
> Thanks,
>
> Di Li
>
>