You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/09/04 15:39:52 UTC
[jira] [Commented] (CLOUDSTACK-7143) Refactor systemvm build
scripts to be easier to test
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121343#comment-14121343 ]
ASF GitHub Bot commented on CLOUDSTACK-7143:
--------------------------------------------
GitHub user lsimons opened a pull request:
https://github.com/apache/cloudstack/pull/16
CLOUDSTACK-7143: Refactoring of the systemvm build process
E-mail thread:
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E
This started out as wanting the systemvm build to take
systemvm/patches/debian/{debian,vpn} from the local machine/branch,
rather than downloading from the apache git master [1]. In working out
how on earth to get veewee to do that cleanly (hint: you can’t, hence
resorting to shar usage) I got quite frustrated with the image rebuild
times.
It so happens that veewee has a --skip-to-postinstall instruction which
is _quite_ useful while debugging these scripts. To get that working
requires the post install steps to be retryable/convergent. Of course,
our existing scripts weren’t set up for that. So I had to add a bunch
of tests whether changes had applied already. Which implied a pretty
significant refactor.
Summarizing this kind of thing is always hard...it’s many little
things...the interesting stuff is at the end/bottom, in particular
the two main improvements
https://github.com/schubergphilis/cloudstack/commit/142d087f6a97f6ac70a858a35d2fe8b638c58cbb
When working on the systemvm in isolation, or using vagrant or
similar tools, it can be useful to inject a custom SSH key before
merging a management server systemvm.iso into it. This option
allows that. It should _not_ have effect on management-server-
managed vms which always get their SSH keys injected.
https://github.com/schubergphilis/cloudstack/commit/e2240eaed18000d4d94dbf6a6e40612db1aeda34
The current build downloads its script from master by fetching a
cloudstack tarball. Besides being an unneeded load on the apache
git server, this is a problem when working on a branch and
wanting to inject a different set of scripts. It also makes it
pretty likely that the injected copy of the script will not match
what a production release wants, so there is very little chance of
not needing to overwrite the scripts.
Ideally we would just rsync over some files. However, veewee does
not provide an option to do that. In order to keep a 'cleanly
veewee-only' build possible, and work with any recent veewee
version, in this change we restor to using shar
(http://en.wikipedia.org/wiki/Shar) to produce an archive which
can execute as a script, which we feed to veewee to execute.
In order to avoid having to re-do this cleanup twice, I also ended up
merging the systemvm and systemvm64 template definitions, factoring out
their small differences by inspecting the os architecture.
https://github.com/schubergphilis/cloudstack/commit/f570b3921cd52672f841fc5f99cdd96f9737d629
https://github.com/schubergphilis/cloudstack/commit/50e91217f90fc952182dccac02a5af06ac33fb45
Everything else…well it pretty much falls into two categories:
* general code cleanup without functional changes
* general code defensiveness to survive various jenkins build
scenarios
All in all it should help with ongoing maintenance, I think.
Most of these commits are now a while old but I wanted to wait with
sending this upstream until we had sufficiently tested the systemvms
built with this changed approach locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/schubergphilis/cloudstack feature/systemvm-refactor-for-upstream
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/16.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16
----
commit 49cbb4e20b70a19286c626ac8d9111be8752bc3f
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T07:54:13Z
CLOUDSTACK-7143: upgrade systemvm to latest debian stable, 7.6.0.
commit 35b1875226201d5923dea57db64bbd789d9ad908
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T07:55:37Z
CLOUDSTACK-7143: split base.sh into its two functions.
commit fb258b506964ca339ef85aa92a6217dc12259811
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T07:57:49Z
CLOUDSTACK-7143: move network tuning from cleanup.sh to its own script.
commit 50e2c0177b0df04aa80d19264d52c8d55dc02eb3
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:36:11Z
CLOUDSTACK-7143: merge systemvm templates, step 1: remove differences
commit 03dba3d901997619d9095e33afe4b39a423ac3f6
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:44:38Z
CLOUDSTACK-7143: merge systemvm templates, step 2: only one template remains
commit afe696edc32dfcaf7eadcb8272f5e1fe31dc32a8
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:52:01Z
CLOUDSTACK-7143: remove 'manual' from preseed config. No other changes.
commit d7aca05d1a728ffdcecf635cf5e339038ec964e9
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:54:57Z
CLOUDSTACK-7143: reformat definition.rb code. No other changes.
commit 6b3657744ee14118bd8c1c72c7154b936de2298d
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:57:32Z
CLOUDSTACK-7143: move add_backports to apt_upgrade.sh
commit 751a4d20ba31386f70987e10059328c718bf1d6e
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T08:58:57Z
CLOUDSTACK-7143: cleanup apt_upgrade.sh code
commit 6129113983a7f5c8c91e98f63d567469d1fcdd22
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:01:15Z
CLOUDSTACK-7143: add vagrant_box_build_time file that's standard with current veewee templates
commit 58a11d37f6441807a9cf65b7440910ec780fc5ee
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:01:56Z
CLOUDSTACK-7143: cleanup configure_gruh.sh code
Adding a check whether GRUB_TIMEOUT has already been set, and if it has,
skip re-running update-grub.
commit 82dedcc0f6fc085961409f55f23cfb568f06558a
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:09:41Z
CLOUDSTACK-7143: split login config out from postinstall.sh
commit 3bf9ddf208593a95843ebb728b735bdb32f91ef0
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:10:59Z
CLOUDSTACK-7143: cleanup configure_login.sh code
Had to change various things to make this code re-entrant. In particular,
the sed-based manipulation of /etc/sudoers is gone and replaced with a
simpler, minimal (but compatible) sudoers file.
Remove the sshd_config tuning since sshd_config is overwritten when we
apply the cloud_scripts overlay (from build.sh).
commit ae65bf132457313b6ec826bbfde78f939337ab88
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:15:23Z
CLOUDSTACK-7143: split locale config into its own file
commit d87d60a1a8dfd887d4b37f23e89cee8354538d0a
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:16:48Z
CLOUDSTACK-7143: cleanup configure_locale.sh code
commit 77677a3318f14fb51cdf4574717269adeaef906e
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:19:03Z
CLOUDSTACK-7143: move fix_acpid to its own file
commit db8eb125caef9b18e3954d70ba024a426651c085
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:19:48Z
CLOUDSTACK-7143: cleanup configure_acpid.sh code
commit 5003845b77abb97ba63d3cd596b0e946d4532411
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:21:03Z
CLOUDSTACK-7143: cleanup cleanup.sh code
commit 6e714dc5cd231c9a96b62e304d292017a33be25c
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:27:16Z
CLOUDSTACK-7143: split connection tracking config into its own file
commit 0addd5a1bf0f6810e901f2b4261c4c5a0ec113e6
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:41:37Z
CLOUDSTACK-7143: move networking logic from postinstall.sh to configure_networking.sh
commit fc87fc4a9ed0b536789500eba573156cadfc6383
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:42:27Z
CLOUDSTACK-7143: cleanup configure_networking.sh code
commit 95d77c2a4674b09546c436beeb1fdef2534a1eee
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T09:46:02Z
CLOUDSTACK-7143: move install_packages from postinstall.sh to its own file
commit cdb30cc59d0bbe90c03d6a1a56c93570f97f2947
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T10:02:13Z
CLOUDSTACK-7143: consider vhdutil a software package
commit 0e62aab3507e43eaf7598bf3f41f76cced97524e
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T10:07:53Z
CLOUDSTACK-7143: reorg of install_systemvm_packages.sh (no logic changes, yet)
commit 3ac947fa36cd20a9a7bcd9f467aff53f9af71ace
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T10:17:06Z
CLOUDSTACK-7143: collapse apt-get install onto one line
In theory this _could_ have changed behavior (apt coming up with a different
solution, or one of the packages configuring a new apt repository), but in my
testing, the end result is the same.
commit 207c50aa20b92a99783fcedb0ac51fb2ec112de6
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T10:28:32Z
CLOUDSTACK-7143: rename postinstall.sh to represent what remains
commit b9f606b4f9ab8943533a7b6d3ca43fecc3cdf5b2
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T11:03:56Z
CLOUDSTACK-7143: cleanup configure_systemvm_services.sh code
commit 30fe938af9e958972bceaea60275c6b2dfada848
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T11:06:34Z
CLOUDSTACK-7143: tune timeouts to expect about <=~60 min build
commit 15673808c7c9500b44e715a3f28f8b21aba0311a
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T11:08:35Z
CLOUDSTACK-7143: cleanup zerodisk.sh code
commit 44a888aa1a9d8f5a14699b87091ef2e34520208e
Author: Leo Simons <ls...@schubergphilis.com>
Date: 2014-07-21T11:09:14Z
CLOUDSTACK-7143: cleanup configure_conntrack.sh code
----
> Refactor systemvm build scripts to be easier to test
> ----------------------------------------------------
>
> Key: CLOUDSTACK-7143
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7143
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: SystemVM
> Reporter: Leo Simons
> Fix For: Future
>
>
> The veewee-wrapping build code could do with some love.
> E-mail thread: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201407.mbox/%3C7A6CF878-7A28-4D4A-BCD2-0C264F8C90B7%40schubergphilis.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)