You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Alex Huang <Al...@citrix.com> on 2014/05/07 02:03:51 UTC
[PROPOSAL] Using continuous integration to maintain our code
quality...
Hi All,
This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
What have we done:
- We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
- We will run the new BVT on master and the current release branch on a continuous basis.
- Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
Please have a read and let me know what you think.
--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Koushik Das <ko...@citrix.com>.
On 14-May-2014, at 3:33 AM, Amogh Vasekar <am...@citrix.com> wrote:
> Hi,
>
> On 5/13/14 1:04 AM, "Daan Hoogland" <da...@gmail.com> wrote:
>
>> If I understood Alex' page on the wiki correctly you can run against
>> your own fork.
>
> I think Non-committers can't have a feature branch on git (at least I
> can't create one)
>
Non-committers can create a branch in github and run against that.
> Thanks,
> Amogh
>
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Hi,
It seems several of my emails to this topics bounced. I added a Q&A section to the original wiki to answer the questions raised in this thread so everyone who reads the process later on can have better understanding. Specifically, the following questions:
Q: I'm a contributor and not a committer. How do I participate in this process?
A: We will attempt to add a way to pull the source code from github as well. In the mean time, you should submit your changes as a patch via the review board. A reviewer will start a branch, apply the patch, wait for CI to pass, and merge it in for you.
Q: Who is providing this jenkins?
A: Currently, the jenkins instance will be provided by Citrix, mainly because we're still working out an infrastructure testing lab in ASF. When that is ready, Citrix is willing to donate the setup necessary to run continuous integration to ASF.
Q: I like to set this up in my own organization so that we can run CI on our own development branches. Can I do this?
A: The setup itself is publicly available. The people who made this possible: Amogh, Bharat, Santhosh, are all on the mailing list. You can communicate with them via the mailing list and ask them help you with the setup. We will try to document this setup as much as possible once it's available.
Q: I currently have a Jenkins slave that runs tests for my own code on every build. Can you add my slave to your jenkins?
A: Currently, no. There's a lot of issues involving jenkins in Citrix making call to outside of Citrix that really I'm not willing to go resolve. For now, you just have to run that with the ASF jenkins.
--Alex
> -----Original Message-----
> From: Amogh Vasekar [mailto:amogh.vasekar@citrix.com]
> Sent: Tuesday, May 13, 2014 3:08 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Sorry, I take that back. Misunderstood the "fork" to be "feature" branch on
> cloudstack git.
>
> Amogh
>
> On 5/13/14 3:03 PM, "Amogh Vasekar" <am...@citrix.com> wrote:
>
> >Hi,
> >
> >On 5/13/14 1:04 AM, "Daan Hoogland" <da...@gmail.com> wrote:
> >
> >>If I understood Alex' page on the wiki correctly you can run against
> >>your own fork.
> >
> >I think Non-committers can't have a feature branch on git (at least I
> >can't create one)
> >
> >Thanks,
> >Amogh
> >
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Amogh Vasekar <am...@citrix.com>.
Sorry, I take that back. Misunderstood the "fork" to be "feature" branch
on cloudstack git.
Amogh
On 5/13/14 3:03 PM, "Amogh Vasekar" <am...@citrix.com> wrote:
>Hi,
>
>On 5/13/14 1:04 AM, "Daan Hoogland" <da...@gmail.com> wrote:
>
>>If I understood Alex' page on the wiki correctly you can run against
>>your own fork.
>
>I think Non-committers can't have a feature branch on git (at least I
>can't create one)
>
>Thanks,
>Amogh
>
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Amogh Vasekar <am...@citrix.com>.
Hi,
On 5/13/14 1:04 AM, "Daan Hoogland" <da...@gmail.com> wrote:
>If I understood Alex' page on the wiki correctly you can run against
>your own fork.
I think Non-committers can't have a feature branch on git (at least I
can't create one)
Thanks,
Amogh
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Actually, this email was sent twice and I answered the first time it was answered. The short answer is yes, everyone can request the CI to be run on their own branch and they must do that before merging it into the release/master branch. Here's my answer enclosed in case it was somehow missed.
" Hi Ritu,
Citrix will provide a Jenkins server that's public accessible that enables you to request BVT to be run on your branch. You don't need to install one yourself. When ASF infra is ready with their hardware, the scripts can be donated to ASF and then developers will need to switch over to the Jenkins operated by ASF.
As for contributors, I did leave out one important part. We will try to allow for the code to be pulled from github but that might come after CI is made available. For now, you'll have to submit your patch through review board and let the reviewer put it through CI for you.
"
--Alex
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Tuesday, May 13, 2014 1:04 AM
> To: dev
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Ritu,
>
> If I understood Alex' page on the wiki correctly you can run against your own
> fork.
> @Alex: true?
>
> On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com>
> wrote:
> > Hi Alex,
> >
> > I am a new developer (non-commiter) and getting to learn about the
> development process of CloudStack.
> >
> > I have a question about the Jenkins, when you say create a branch for your
> code and ask Jenkins to run BVT on your branch. The branch will be created
> on my local repository but Jenkins would run on central repository. In that
> case, does it mean that I install Jenkins locally on my setup.
> >
> > Please help.
> >
> > Thanks,
> > Ritu S.
> >
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Tuesday, May 06, 2014 5:04 PM
> > To: dev@cloudstack.apache.org
> > Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
> >
> > Hi All,
> >
> > This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
> >
> > This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
> >
> > What have we done:
> > - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> > - We will run the new BVT on master and the current release branch on a
> continuous basis.
> > - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
> >
> > Please have a read and let me know what you think.
> >
> > --Alex
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> ro
> > cess
>
>
>
> --
> Daan
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Santhosh Edukulla <sa...@citrix.com>.
We have put up a wiki(initial cut\draft), will add more information once time permits
https://cwiki.apache.org/confluence/display/CLOUDSTACK/New+Infrastructure,CI,Simulator,Automation+Changes
Thanks!
Santhosh
________________________________________
From: Alex Huang [Alex.Huang@citrix.com]
Sent: Thursday, May 15, 2014 5:01 PM
To: dev@cloudstack.apache.org
Subject: RE: [PROPOSAL] Using continuous integration to maintain our code quality...
> On Wed, May 14, 2014 at 06:21:34PM +0000, Alex Huang wrote:
> > I think infrastructure code should just be checked in with the source
> > code. To separate it means you have to deal with version
> > match/mismatch between infrastructure and source code.
>
> Sorry - that doesn't sound right. Usually infrastructure code has little to do
> with the source code other than deploying the packages built from said
> source. Could you please clarify the bits that go into this infra code and why it
> should be affected by versions? [*] If it's config management recipes those
> are usually maintained separate from source of your web-tier. Infra code is
> maintained and changed usually more rapidly. I don't get why jenkins
> settings and config management should exist within our catch-all tools dir.
> We're going to bloat it up unnecessarily and realise later it should have been
> a separate repo to begin with - for eg. cloudmonkey or marvin.
Let me put out what I consider as a requirement. The requirement is any changes in framework, infrastructure, configuration, recipes, scripts, source code, marvin, etc, cannot affect CI that has been established on released branches. You have to look at CI like build for source code. When a release is done, you take a label or a branch and you're fairly certain it will build again. CI has to be the same, I might have shut it down for a release for a while but if I have to dust it off, I have to be fairly certain it runs again without a lot of debugging. Now if there are well established components, like Jenkins or Mysql that we can just specify the version # like we do with maven in build, then it's fine not to include it in the source tree. But if the components used by CI is not a well versioned independent entity, then we should include it in the source tree and branch it with the release or risk CI breaking for that release. For things like this, I rather not guess too optimistically about the chances. We have to treat CI running on released branches to be like production systems. The best thing to do for a production system that works is to don't let anyone touch any part of it, as any ops guy will tell you.
>
> [*] the wiki did not have enough information about the infra-code
It's not intended to. The wiki is meant to provide what developers and testers should do. It's not to explain the infrastructure code. I'm sure Santhosh and others will document what they've done and how others can take advantage.
--Alex
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Prasanna Santhanam <ts...@apache.org>.
On Thu, May 15, 2014 at 09:01:16PM +0000, Alex Huang wrote:
> > On Wed, May 14, 2014 at 06:21:34PM +0000, Alex Huang wrote:
> > > I think infrastructure code should just be checked in with the source
> > > code. To separate it means you have to deal with version
> > > match/mismatch between infrastructure and source code.
> >
> > Sorry - that doesn't sound right. Usually infrastructure code has little to do
> > with the source code other than deploying the packages built from said
> > source. Could you please clarify the bits that go into this infra code and why it
> > should be affected by versions? [*] If it's config management recipes those
> > are usually maintained separate from source of your web-tier. Infra code is
> > maintained and changed usually more rapidly. I don't get why jenkins
> > settings and config management should exist within our catch-all tools dir.
> > We're going to bloat it up unnecessarily and realise later it should have been
> > a separate repo to begin with - for eg. cloudmonkey or marvin.
>
> Let me put out what I consider as a requirement. The requirement is
> any changes in framework, infrastructure, configuration, recipes,
> scripts, source code, marvin, etc, cannot affect CI that has been
> established on released branches. You have to look at CI like build
> for source code. When a release is done, you take a label or a
> branch and you're fairly certain it will build again. CI has to be
> the same, I might have shut it down for a release for a while but if
> I have to dust it off, I have to be fairly certain it runs again
> without a lot of debugging. Now if there are well established
> components, like Jenkins or Mysql that we can just specify the
> version # like we do with maven in build, then it's fine not to
> include it in the source tree. But if the components used by CI is
> not a well versioned independent entity, then we should include it
> in the source tree and branch it with the release or risk CI
> breaking for that release. For things like this, I rather not guess
> too optimistically about the chances. We have to treat CI running
> on released branches to be like production systems. The best thing
> to do for a production system that works is to don't let anyone
> touch any part of it, as any ops guy will tell you.
Ok - easier to understand this when you guys are ready to share your
framework/tools. Even so, I forsee it being composed of tools and
libraries that have little to nothing to do with CloudStack. Which is
why I would like it to be part of a different repo, have their own
independant releases so infra code can be deployed and upgraded by ops
guys (of various companies using the solution) at the time they see
best. For instance - jenkins instances upgrade regardless of the
projects they are building.
>
> >
> > [*] the wiki did not have enough information about the infra-code
>
> It's not intended to. The wiki is meant to provide what developers
> and testers should do. It's not to explain the infrastructure code.
> I'm sure Santhosh and others will document what they've done and how
> others can take advantage.
Look forward to it.
>
> --Alex
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
> On Wed, May 14, 2014 at 06:21:34PM +0000, Alex Huang wrote:
> > I think infrastructure code should just be checked in with the source
> > code. To separate it means you have to deal with version
> > match/mismatch between infrastructure and source code.
>
> Sorry - that doesn't sound right. Usually infrastructure code has little to do
> with the source code other than deploying the packages built from said
> source. Could you please clarify the bits that go into this infra code and why it
> should be affected by versions? [*] If it's config management recipes those
> are usually maintained separate from source of your web-tier. Infra code is
> maintained and changed usually more rapidly. I don't get why jenkins
> settings and config management should exist within our catch-all tools dir.
> We're going to bloat it up unnecessarily and realise later it should have been
> a separate repo to begin with - for eg. cloudmonkey or marvin.
Let me put out what I consider as a requirement. The requirement is any changes in framework, infrastructure, configuration, recipes, scripts, source code, marvin, etc, cannot affect CI that has been established on released branches. You have to look at CI like build for source code. When a release is done, you take a label or a branch and you're fairly certain it will build again. CI has to be the same, I might have shut it down for a release for a while but if I have to dust it off, I have to be fairly certain it runs again without a lot of debugging. Now if there are well established components, like Jenkins or Mysql that we can just specify the version # like we do with maven in build, then it's fine not to include it in the source tree. But if the components used by CI is not a well versioned independent entity, then we should include it in the source tree and branch it with the release or risk CI breaking for that release. For things like this, I rather not guess too optimistically about the chances. We have to treat CI running on released branches to be like production systems. The best thing to do for a production system that works is to don't let anyone touch any part of it, as any ops guy will tell you.
>
> [*] the wiki did not have enough information about the infra-code
It's not intended to. The wiki is meant to provide what developers and testers should do. It's not to explain the infrastructure code. I'm sure Santhosh and others will document what they've done and how others can take advantage.
--Alex
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Prasanna Santhanam <ts...@apache.org>.
On Wed, May 14, 2014 at 06:21:34PM +0000, Alex Huang wrote:
> I think infrastructure code should just be checked in with the
> source code. To separate it means you have to deal with version
> match/mismatch between infrastructure and source code.
Sorry - that doesn't sound right. Usually infrastructure code has
little to do with the source code other than deploying the packages
built from said source. Could you please clarify the bits that go into
this infra code and why it should be affected by versions? [*] If it's
config management recipes those are usually maintained separate from
source of your web-tier. Infra code is maintained and changed usually
more rapidly. I don't get why jenkins settings and config management
should exist within our catch-all tools dir. We're going to bloat it
up unnecessarily and realise later it should have been a separate repo
to begin with - for eg. cloudmonkey or marvin.
[*] the wiki did not have enough information about the infra-code
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
I think infrastructure code should just be checked in with the source code. To separate it means you have to deal with version match/mismatch between infrastructure and source code.
--Alex
> -----Original Message-----
> From: Santhosh Edukulla [mailto:santhosh.edukulla@citrix.com]
> Sent: Tuesday, May 13, 2014 10:25 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Infrastructure code related changes done\in progress, related to this
> proposal, currently are maintained in a separate repo.
>
> If this needs to be part of ASF, please let us know a suitable place to commit
> these changes.
>
> Its suggestible that all the infrastructure changes be part of a separate
> branch under CS, so that it is easy to maintain and use. Please let us know, so
> that we can create one and check them in to be made available to all.
>
> Regards,
> Santhosh
> ________________________________________
> From: Alex Huang [Alex.Huang@citrix.com]
> Sent: Tuesday, May 13, 2014 3:43 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Noji,
>
> Everything should be checked in under tools. I'll make sure of that. None of
> the code/configuration should be private.
>
> --Alex
>
> > -----Original Message-----
> > From: ynojima@ynojima.net [mailto:ynojima@ynojima.net] On Behalf Of
> > Yoshikazu Nojima
> > Sent: Tuesday, May 13, 2014 8:20 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] Using continuous integration to maintain our
> > code quality...
> >
> > Alex,
> >
> > Thank you for interesting proposal. I suppose pre check-in test is
> > long- awaited for everyone.
> > I'm looking forward to see your Jenkins made public.
> >
> > If possible, can you share jenkins job's settings.xml for future
> > reference when it is ready?
> >
> > Regards,
> > Noji
> >
> > 2014-05-13 2:04 GMT-06:00 Daan Hoogland <da...@gmail.com>:
> > > Ritu,
> > >
> > > If I understood Alex' page on the wiki correctly you can run against
> > > your own fork.
> > > @Alex: true?
> > >
> > > On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal
> > > <rs...@brocade.com>
> > wrote:
> > >> Hi Alex,
> > >>
> > >> I am a new developer (non-commiter) and getting to learn about the
> > development process of CloudStack.
> > >>
> > >> I have a question about the Jenkins, when you say create a branch
> > >> for
> > your code and ask Jenkins to run BVT on your branch. The branch will
> > be created on my local repository but Jenkins would run on central
> > repository. In that case, does it mean that I install Jenkins locally on my
> setup.
> > >>
> > >> Please help.
> > >>
> > >> Thanks,
> > >> Ritu S.
> > >>
> > >> -----Original Message-----
> > >> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > >> Sent: Tuesday, May 06, 2014 5:04 PM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: [PROPOSAL] Using continuous integration to maintain our
> > >> code
> > quality...
> > >>
> > >> Hi All,
> > >>
> > >> This is something I brought up a long time ago but really didn't
> > >> have the
> > resources to get it all up and running until now. Throughout the past
> > year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others
> > have been chipping away at it. At this point, we finally pull
> > together a continuous integration setup that we can use to make sure
> > that CloudStack master and the currently release branch are always
> > stable. This is getting pretty close to be completed and we like to
> > share it with the community in hopes that we can reduce/eliminate that
> problems we've seen with our recent releases.
> > Currently, the physical hardware are hosted by Citrix but we'll be
> > more than willing to donate the work to infra when that's all settled.
> > >>
> > >> This does require effort from the community to make a change in
> > >> their
> > development process. These steps are detailed at [1]. I like to get
> > feedback on what everyone think about this.
> > >>
> > >> What have we done:
> > >> - We replaced a large selection of the BVT tests to run with the
> > >> simulator
> > instead of actual hardware. This shortens the duration of each BVT run.
> > Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> > minutes.
> > >> - We will run the new BVT on master and the current release
> > >> branch on a
> > continuous basis.
> > >> - Developers can use Jenkins to ask BVT to be run on their branch
> > >> so they
> > can know it won't break the continuous integration before they merge
> > to master and the current release branch.
> > >>
> > >> Please have a read and let me know what you think.
> > >>
> > >> --Alex
> > >>
> > >> [1]
> > >>
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> > r
> > >> ocess
> > >
> > >
> > >
> > > --
> > > Daan
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Prasanna Santhanam <ts...@apache.org>.
https://github.com/CloudStack-extras has been used in the past but
fell out of use. If part of the ASF then it will go through the
regular committer bits and process which users contributing to infra
*may* see as overhead.
On Wed, May 14, 2014 at 05:25:19AM +0000, Santhosh Edukulla wrote:
> Infrastructure code related changes done\in progress, related to
> this proposal, currently are maintained in a separate repo.
>
> If this needs to be part of ASF, please let us know a suitable place
> to commit these changes.
>
> Its suggestible that all the infrastructure changes be part of a
> separate branch under CS, so that it is easy to maintain and use.
> Please let us know, so that we can create one and check them in to
> be made available to all.
>
> Regards,
> Santhosh
> ________________________________________
> From: Alex Huang [Alex.Huang@citrix.com]
> Sent: Tuesday, May 13, 2014 3:43 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] Using continuous integration to maintain our code quality...
>
> Noji,
>
> Everything should be checked in under tools. I'll make sure of that. None of the code/configuration should be private.
>
> --Alex
>
> > -----Original Message-----
> > From: ynojima@ynojima.net [mailto:ynojima@ynojima.net] On Behalf Of
> > Yoshikazu Nojima
> > Sent: Tuesday, May 13, 2014 8:20 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> > quality...
> >
> > Alex,
> >
> > Thank you for interesting proposal. I suppose pre check-in test is long-
> > awaited for everyone.
> > I'm looking forward to see your Jenkins made public.
> >
> > If possible, can you share jenkins job's settings.xml for future reference
> > when it is ready?
> >
> > Regards,
> > Noji
> >
> > 2014-05-13 2:04 GMT-06:00 Daan Hoogland <da...@gmail.com>:
> > > Ritu,
> > >
> > > If I understood Alex' page on the wiki correctly you can run against
> > > your own fork.
> > > @Alex: true?
> > >
> > > On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com>
> > wrote:
> > >> Hi Alex,
> > >>
> > >> I am a new developer (non-commiter) and getting to learn about the
> > development process of CloudStack.
> > >>
> > >> I have a question about the Jenkins, when you say create a branch for
> > your code and ask Jenkins to run BVT on your branch. The branch will be
> > created on my local repository but Jenkins would run on central repository. In
> > that case, does it mean that I install Jenkins locally on my setup.
> > >>
> > >> Please help.
> > >>
> > >> Thanks,
> > >> Ritu S.
> > >>
> > >> -----Original Message-----
> > >> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > >> Sent: Tuesday, May 06, 2014 5:04 PM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: [PROPOSAL] Using continuous integration to maintain our code
> > quality...
> > >>
> > >> Hi All,
> > >>
> > >> This is something I brought up a long time ago but really didn't have the
> > resources to get it all up and running until now. Throughout the past year,
> > Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> > chipping away at it. At this point, we finally pull together a continuous
> > integration setup that we can use to make sure that CloudStack master and
> > the currently release branch are always stable. This is getting pretty close to
> > be completed and we like to share it with the community in hopes that we
> > can reduce/eliminate that problems we've seen with our recent releases.
> > Currently, the physical hardware are hosted by Citrix but we'll be more than
> > willing to donate the work to infra when that's all settled.
> > >>
> > >> This does require effort from the community to make a change in their
> > development process. These steps are detailed at [1]. I like to get feedback
> > on what everyone think about this.
> > >>
> > >> What have we done:
> > >> - We replaced a large selection of the BVT tests to run with the simulator
> > instead of actual hardware. This shortens the duration of each BVT run.
> > Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> > minutes.
> > >> - We will run the new BVT on master and the current release branch on a
> > continuous basis.
> > >> - Developers can use Jenkins to ask BVT to be run on their branch so they
> > can know it won't break the continuous integration before they merge to
> > master and the current release branch.
> > >>
> > >> Please have a read and let me know what you think.
> > >>
> > >> --Alex
> > >>
> > >> [1]
> > >>
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> > r
> > >> ocess
> > >
> > >
> > >
> > > --
> > > Daan
--
Prasanna.,
------------------------
Powered by BigRock.com
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Santhosh Edukulla <sa...@citrix.com>.
Infrastructure code related changes done\in progress, related to this proposal, currently are maintained in a separate repo.
If this needs to be part of ASF, please let us know a suitable place to commit these changes.
Its suggestible that all the infrastructure changes be part of a separate branch under CS, so that it is easy to maintain and use. Please let us know, so that we can create one and check them in to be made available to all.
Regards,
Santhosh
________________________________________
From: Alex Huang [Alex.Huang@citrix.com]
Sent: Tuesday, May 13, 2014 3:43 PM
To: dev@cloudstack.apache.org
Subject: RE: [PROPOSAL] Using continuous integration to maintain our code quality...
Noji,
Everything should be checked in under tools. I'll make sure of that. None of the code/configuration should be private.
--Alex
> -----Original Message-----
> From: ynojima@ynojima.net [mailto:ynojima@ynojima.net] On Behalf Of
> Yoshikazu Nojima
> Sent: Tuesday, May 13, 2014 8:20 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Alex,
>
> Thank you for interesting proposal. I suppose pre check-in test is long-
> awaited for everyone.
> I'm looking forward to see your Jenkins made public.
>
> If possible, can you share jenkins job's settings.xml for future reference
> when it is ready?
>
> Regards,
> Noji
>
> 2014-05-13 2:04 GMT-06:00 Daan Hoogland <da...@gmail.com>:
> > Ritu,
> >
> > If I understood Alex' page on the wiki correctly you can run against
> > your own fork.
> > @Alex: true?
> >
> > On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com>
> wrote:
> >> Hi Alex,
> >>
> >> I am a new developer (non-commiter) and getting to learn about the
> development process of CloudStack.
> >>
> >> I have a question about the Jenkins, when you say create a branch for
> your code and ask Jenkins to run BVT on your branch. The branch will be
> created on my local repository but Jenkins would run on central repository. In
> that case, does it mean that I install Jenkins locally on my setup.
> >>
> >> Please help.
> >>
> >> Thanks,
> >> Ritu S.
> >>
> >> -----Original Message-----
> >> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> >> Sent: Tuesday, May 06, 2014 5:04 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
> >>
> >> Hi All,
> >>
> >> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
> >>
> >> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
> >>
> >> What have we done:
> >> - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> >> - We will run the new BVT on master and the current release branch on a
> continuous basis.
> >> - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
> >>
> >> Please have a read and let me know what you think.
> >>
> >> --Alex
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> r
> >> ocess
> >
> >
> >
> > --
> > Daan
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Noji,
Everything should be checked in under tools. I'll make sure of that. None of the code/configuration should be private.
--Alex
> -----Original Message-----
> From: ynojima@ynojima.net [mailto:ynojima@ynojima.net] On Behalf Of
> Yoshikazu Nojima
> Sent: Tuesday, May 13, 2014 8:20 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Alex,
>
> Thank you for interesting proposal. I suppose pre check-in test is long-
> awaited for everyone.
> I'm looking forward to see your Jenkins made public.
>
> If possible, can you share jenkins job's settings.xml for future reference
> when it is ready?
>
> Regards,
> Noji
>
> 2014-05-13 2:04 GMT-06:00 Daan Hoogland <da...@gmail.com>:
> > Ritu,
> >
> > If I understood Alex' page on the wiki correctly you can run against
> > your own fork.
> > @Alex: true?
> >
> > On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com>
> wrote:
> >> Hi Alex,
> >>
> >> I am a new developer (non-commiter) and getting to learn about the
> development process of CloudStack.
> >>
> >> I have a question about the Jenkins, when you say create a branch for
> your code and ask Jenkins to run BVT on your branch. The branch will be
> created on my local repository but Jenkins would run on central repository. In
> that case, does it mean that I install Jenkins locally on my setup.
> >>
> >> Please help.
> >>
> >> Thanks,
> >> Ritu S.
> >>
> >> -----Original Message-----
> >> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> >> Sent: Tuesday, May 06, 2014 5:04 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
> >>
> >> Hi All,
> >>
> >> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
> >>
> >> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
> >>
> >> What have we done:
> >> - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> >> - We will run the new BVT on master and the current release branch on a
> continuous basis.
> >> - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
> >>
> >> Please have a read and let me know what you think.
> >>
> >> --Alex
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> r
> >> ocess
> >
> >
> >
> > --
> > Daan
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Yoshikazu Nojima <ma...@ynojima.net>.
Alex,
Thank you for interesting proposal. I suppose pre check-in test is
long-awaited for everyone.
I'm looking forward to see your Jenkins made public.
If possible, can you share jenkins job's settings.xml for future
reference when it is ready?
Regards,
Noji
2014-05-13 2:04 GMT-06:00 Daan Hoogland <da...@gmail.com>:
> Ritu,
>
> If I understood Alex' page on the wiki correctly you can run against
> your own fork.
> @Alex: true?
>
> On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com> wrote:
>> Hi Alex,
>>
>> I am a new developer (non-commiter) and getting to learn about the development process of CloudStack.
>>
>> I have a question about the Jenkins, when you say create a branch for your code and ask Jenkins to run BVT on your branch. The branch will be created on my local repository but Jenkins would run on central repository. In that case, does it mean that I install Jenkins locally on my setup.
>>
>> Please help.
>>
>> Thanks,
>> Ritu S.
>>
>> -----Original Message-----
>> From: Alex Huang [mailto:Alex.Huang@citrix.com]
>> Sent: Tuesday, May 06, 2014 5:04 PM
>> To: dev@cloudstack.apache.org
>> Subject: [PROPOSAL] Using continuous integration to maintain our code quality...
>>
>> Hi All,
>>
>> This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
>>
>> This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
>>
>> What have we done:
>> - We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
>> - We will run the new BVT on master and the current release branch on a continuous basis.
>> - Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
>>
>> Please have a read and let me know what you think.
>>
>> --Alex
>>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
>
>
>
> --
> Daan
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Daan Hoogland <da...@gmail.com>.
Ritu,
If I understood Alex' page on the wiki correctly you can run against
your own fork.
@Alex: true?
On Wed, May 7, 2014 at 8:07 PM, Ritu Sabharwal <rs...@brocade.com> wrote:
> Hi Alex,
>
> I am a new developer (non-commiter) and getting to learn about the development process of CloudStack.
>
> I have a question about the Jenkins, when you say create a branch for your code and ask Jenkins to run BVT on your branch. The branch will be created on my local repository but Jenkins would run on central repository. In that case, does it mean that I install Jenkins locally on my setup.
>
> Please help.
>
> Thanks,
> Ritu S.
>
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Tuesday, May 06, 2014 5:04 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] Using continuous integration to maintain our code quality...
>
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
> - We will run the new BVT on master and the current release branch on a continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
--
Daan
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Ritu Sabharwal <rs...@Brocade.com>.
Hi Alex,
I am a new developer (non-commiter) and getting to learn about the development process of CloudStack.
I have a question about the Jenkins, when you say create a branch for your code and ask Jenkins to run BVT on your branch. The branch will be created on my local repository but Jenkins would run on central repository. In that case, does it mean that I install Jenkins locally on my setup.
Please help.
Thanks,
Ritu S.
-----Original Message-----
From: Alex Huang [mailto:Alex.Huang@citrix.com]
Sent: Tuesday, May 06, 2014 5:04 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] Using continuous integration to maintain our code quality...
Hi All,
This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
What have we done:
- We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
- We will run the new BVT on master and the current release branch on a continuous basis.
- Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
Please have a read and let me know what you think.
--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Ritu Sabharwal <rs...@Brocade.com>.
Thanks Alex for the information!
-----Original Message-----
From: Alex Huang [mailto:Alex.Huang@citrix.com]
Sent: Wednesday, May 07, 2014 2:56 PM
To: Ritu Sabharwal
Cc: dev@cloudstack.apache.org
Subject: RE: [PROPOSAL] Using continuous integration to maintain our code quality...
Hi Ritu,
Citrix will provide a Jenkins server that's public accessible that enables you to request BVT to be run on your branch. You don't need to install one yourself. When ASF infra is ready with their hardware, the scripts can be donated to ASF and then developers will need to switch over to the Jenkins operated by ASF.
As for contributors, I did leave out one important part. We will try to allow for the code to be pulled from github but that might come after CI is made available. For now, you'll have to submit your patch through review board and let the reviewer put it through CI for you.
--Alex
> -----Original Message-----
> From: Ritu Sabharwal [mailto:rsabharw@Brocade.com]
> Sent: Wednesday, May 7, 2014 11:23 AM
> To: Alex Huang
> Cc: dev@cloudstack.apache.org
> Subject: FW: [PROPOSAL] Using continuous integration to maintain our
> code quality...
>
> Hi Alex,
>
> I am a new developer (non-commiter) and getting to learn about the
> development process of CloudStack.
>
> I have a question about the Jenkins, when you say create a branch for
> your code and ask Jenkins to run BVT on your branch. The branch will
> be created on my local repository but Jenkins would run on central
> repository. In that case, does it mean that I install Jenkins locally
> on my setup or do I o need to create account on Jenkins and run BVT on my local branch.
>
> Please help.
>
> Thanks,
> Ritu S.
>
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Tuesday, May 06, 2014 5:04 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Hi All,
>
> This is something I brought up a long time ago but really didn't have
> the resources to get it all up and running until now. Throughout the
> past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and
> others have been chipping away at it. At this point, we finally pull
> together a continuous integration setup that we can use to make sure
> that CloudStack master and the currently release branch are always
> stable. This is getting pretty close to be completed and we like to
> share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be
> more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get
> feedback on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the
> simulator instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> - We will run the new BVT on master and the current release branch
> on a continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so
> they can know it won't break the continuous integration before they
> merge to master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> rocess
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Hi Ritu,
Citrix will provide a Jenkins server that's public accessible that enables you to request BVT to be run on your branch. You don't need to install one yourself. When ASF infra is ready with their hardware, the scripts can be donated to ASF and then developers will need to switch over to the Jenkins operated by ASF.
As for contributors, I did leave out one important part. We will try to allow for the code to be pulled from github but that might come after CI is made available. For now, you'll have to submit your patch through review board and let the reviewer put it through CI for you.
--Alex
> -----Original Message-----
> From: Ritu Sabharwal [mailto:rsabharw@Brocade.com]
> Sent: Wednesday, May 7, 2014 11:23 AM
> To: Alex Huang
> Cc: dev@cloudstack.apache.org
> Subject: FW: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Hi Alex,
>
> I am a new developer (non-commiter) and getting to learn about the
> development process of CloudStack.
>
> I have a question about the Jenkins, when you say create a branch for your
> code and ask Jenkins to run BVT on your branch. The branch will be created
> on my local repository but Jenkins would run on central repository. In that
> case, does it mean that I install Jenkins locally on my setup or do I o need to
> create account on Jenkins and run BVT on my local branch.
>
> Please help.
>
> Thanks,
> Ritu S.
>
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Tuesday, May 06, 2014 5:04 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> - We will run the new BVT on master and the current release branch on a
> continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> rocess
FW: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Ritu Sabharwal <rs...@Brocade.com>.
Hi Alex,
I am a new developer (non-commiter) and getting to learn about the development process of CloudStack.
I have a question about the Jenkins, when you say create a branch for your code and ask Jenkins to run BVT on your branch. The branch will be created on my local repository but Jenkins would run on central repository. In that case, does it mean that I install Jenkins locally on my setup or do I o need to create account on Jenkins and run BVT on my local branch.
Please help.
Thanks,
Ritu S.
-----Original Message-----
From: Alex Huang [mailto:Alex.Huang@citrix.com]
Sent: Tuesday, May 06, 2014 5:04 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] Using continuous integration to maintain our code quality...
Hi All,
This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
What have we done:
- We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
- We will run the new BVT on master and the current release branch on a continuous basis.
- Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
Please have a read and let me know what you think.
--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Erik Weber [mailto:terbolous@gmail.com]
> Sent: Wednesday, August 20, 2014 12:25 PM
> To: dev
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> On Wed, May 7, 2014 at 2:03 AM, Alex Huang <Al...@citrix.com>
> wrote:
>
> > Hi All,
> >
> > This is something I brought up a long time ago but really didn't have
> > the resources to get it all up and running until now. Throughout the
> > past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and
> > others have been chipping away at it. At this point, we finally pull
> > together a continuous integration setup that we can use to make sure
> > that CloudStack master and the currently release branch are always
> > stable. This is getting pretty close to be completed and we like to
> > share it with the community in hopes that we can reduce/eliminate that
> > problems we've seen with our recent releases. Currently, the physical
> > hardware are hosted by Citrix but we'll be more than willing to donate the
> work to infra when that's all settled.
> >
> > This does require effort from the community to make a change in their
> > development process. These steps are detailed at [1]. I like to get
> > feedback on what everyone think about this.
> >
> > What have we done:
> > - We replaced a large selection of the BVT tests to run with the
> > simulator instead of actual hardware. This shortens the duration of
> > each BVT run. Today, a BVT that runs tests for XenServer and KVM
> > completes in
> > 30-40 minutes.
> >
>
> How much is running with Simulator instead of actual hardware? My issue
> with this is that you're testing against a flawless simulator in terms of testing,
> while with actual hardware you are bound to hit bugs/issues that might not
> be due to ACS code but ACS still has to handle it.
>
> As an example, could you run a test on the tags '4.4.0' and '4.3.0' and report
> the result? They both had fundamental flaws, where the one was practically
> useless for a week or so, and the other had major issues with KVM, and if the
> BVT doesn't encounter those because it's using the simulator I see it as a
> burden rather than a gift, since you're relying on a false result.
For KVM, there is another way to test basic KVM features without using real HW:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/devcloud-kvm
Currently, I am working on a machine I created in digtialOcean, and make sure it can
test against real KVM, not only just simulator. If it's working, I can donate it as one of
CI machine.
>
> --
> Erik
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Amogh Vasekar <am...@citrix.com>.
Hi,
On 8/20/14 12:24 PM, "Erik Weber" <te...@gmail.com> wrote:
>They both had fundamental flaws, where the one was
>practically useless for a week or so, and the other had major issues with
>KVM, and if the BVT doesn't encounter those because it's using the
>simulator I see it as a burden rather than a gift, since you're relying on
>a false result.
AFAIK, only the non-resource-layer related tests were moved to Simulator,
the idea being it helps catch bugs at management layer quickly without
needing a full set-up.
Of course, bugs related to actual resource layer may not be exposed by
Simulator runs.
In addition, I believe Koushik had done a lot of work to simulate failure
scenarios etc so that we don't only test the "happy path" using Simulator.
HTH
Amogh
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Erik Weber <te...@gmail.com>.
On Wed, May 7, 2014 at 2:03 AM, Alex Huang <Al...@citrix.com> wrote:
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past
> year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have
> been chipping away at it. At this point, we finally pull together a
> continuous integration setup that we can use to make sure that CloudStack
> master and the currently release branch are always stable. This is getting
> pretty close to be completed and we like to share it with the community in
> hopes that we can reduce/eliminate that problems we've seen with our recent
> releases. Currently, the physical hardware are hosted by Citrix but we'll
> be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get
> feedback on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the
> simulator instead of actual hardware. This shortens the duration of each
> BVT run. Today, a BVT that runs tests for XenServer and KVM completes in
> 30-40 minutes.
>
How much is running with Simulator instead of actual hardware? My issue
with this is that you're testing against a flawless simulator in terms of
testing, while with actual hardware you are bound to hit bugs/issues that
might not be due to ACS code but ACS still has to handle it.
As an example, could you run a test on the tags '4.4.0' and '4.3.0' and
report the result? They both had fundamental flaws, where the one was
practically useless for a week or so, and the other had major issues with
KVM, and if the BVT doesn't encounter those because it's using the
simulator I see it as a burden rather than a gift, since you're relying on
a false result.
--
Erik
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Sebastien Goasguen <ru...@gmail.com>.
On May 6, 2014, at 8:03 PM, Alex Huang <Al...@citrix.com> wrote:
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
>
digression: you are the community
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
> - We will run the new BVT on master and the current release branch on a continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
+1 on this in general.
The wiki you refer to talks about the process. But you mention that you have done it internally, not on apache jenkins. Are there any code changes to marvin or the simulator that have not been committed to our repo yet ?
Overall I think the process is good but probably needs to be clarified between committer and non-committer process. For instance you mention that if the BVT fails the commit will be reverted ? I presume this applies to committers who won't test their own github branch -like you propose for non-committers- ?
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Daan Hoogland <da...@gmail.com>.
Alex, (see inline)
On Mon, May 12, 2014 at 6:02 PM, Alex Huang <Al...@citrix.com> wrote:
...
> For the concern on distributed setup, there's a two part answer.
> 1. We're basically asking everyone to use a central Jenkins to run the automated before they merge to the asf git repository to make sure they didn't break the product for everyone. It shouldn't have any impact on your local Jenkins.
I am not talking of a local jenkins but of a local jenkins slave
machine triggered by the central jenkins. I know solidfire has their
own setup with their own hardware that only they can test their plugin
code agains and they want to make sure new checkins don't break their
code. We want such a setup for more then just their code/hardware. So
if someone triggers a check it should check on several build machines,
the central one and also all other machines that 'subscribed' as a
regression test machine. Am I drifting?
...
> As for reviewers, I'm open to the number of reviewers. The only number I will be against definitely is zero. My personal opinion is that the quality of the reviewer is more important than the quantity. I'm not sure how we can make sure the quality of the reviewer is good. I'm open to suggestions there.
I aggree that the quality is more important then the quantity. To
outside contributers it is the availibility that is worrying.
Daan
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Daan,
I think every part of the proposal is up for discussion and suggestions. It's why I put it on to the list for everyone to review.
For the concern on distributed setup, there's a two part answer.
1. We're basically asking everyone to use a central Jenkins to run the automated before they merge to the asf git repository to make sure they didn't break the product for everyone. It shouldn't have any impact on your local Jenkins. It's perfectly fine for to run all tests yourself for the majority of your code/build/test cycle and only run one sanity test when submitting to the asf git repo. Note that we're only asking this for the master and the currently releasing branch. Your remote branches are yours only.
2. All of the software that we wrote/using are checked into the asf git repo, we welcome anyone to set this up for themselves if you need something more speedy. If you do set it up, then I don't see why you need to use the service that is proposed here.
As for reviewers, I'm open to the number of reviewers. The only number I will be against definitely is zero. My personal opinion is that the quality of the reviewer is more important than the quantity. I'm not sure how we can make sure the quality of the reviewer is good. I'm open to suggestions there.
The page itself is still being updated. Some of the links need to be cover in more detail and some links need to be updated. I will continue to update the list.
--Alex
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, May 7, 2014 3:34 AM
> To: dev
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Alex,
>
> I applaud the effort and everybody at Citrix contributing to it.
> This is not a concern with your proposed process: I hope it does not impede
> the efforts to have a distributed setup where everybody can start their tests
> specific to their hardware from the central jenkins on their local jenkins
> slaves.
> One concern that I do have is the finding of two reviewers. I think we can ask
> of people to put effort into finding two reviewers but it is my experience that
> these are not always available/willing/able to make time.
> The page you made looks good. You could add link in the list of 'developer
> resposibilities' to relevant documentation.
>
> kind regards,
> Daan
>
> On Wed, May 7, 2014 at 2:03 AM, Alex Huang <Al...@citrix.com>
> wrote:
> > Hi All,
> >
> > This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
> >
> > This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
> >
> > What have we done:
> > - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> > - We will run the new BVT on master and the current release branch on a
> continuous basis.
> > - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
> >
> > Please have a read and let me know what you think.
> >
> > --Alex
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> ro
> > cess
>
>
>
> --
> Daan
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Daan Hoogland <da...@gmail.com>.
Alex,
I applaud the effort and everybody at Citrix contributing to it.
This is not a concern with your proposed process: I hope it does not
impede the efforts to have a distributed setup where everybody can
start their tests specific to their hardware from the central jenkins
on their local jenkins slaves.
One concern that I do have is the finding of two reviewers. I think we
can ask of people to put effort into finding two reviewers but it is
my experience that these are not always available/willing/able to make
time.
The page you made looks good. You could add link in the list of
'developer resposibilities' to relevant documentation.
kind regards,
Daan
On Wed, May 7, 2014 at 2:03 AM, Alex Huang <Al...@citrix.com> wrote:
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
> - We will run the new BVT on master and the current release branch on a continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
--
Daan
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Daan Hoogland <da...@gmail.com>.
On Wed, May 7, 2014 at 11:20 AM, Donal Lafferty
<do...@citrix.com> wrote:
> Can we introduce a penalty points system to get them to slow down?
>
> E.g. 3 pts / offence, mandatory 3rd party code reviewers after 15 over two releases cycles?
Penalty will only work in combination with positive incentives. If not
people will just get good at avoiding penalty. Gamification yes, but
it should go two ways.
--
Daan
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Donal Lafferty <do...@citrix.com>.
How about some gamification? :)
Continuous integration allows devs to gage whether they're coding too fast for quality conditions. Can we introduce a penalty points system to get them to slow down?
E.g. 3 pts / offence, mandatory 3rd party code reviewers after 15 over two releases cycles?
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: 07 May 2014 01:04
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> - We will run the new BVT on master and the current release branch on a
> continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+P
> rocess
Re: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Amogh Vasekar <am...@citrix.com>.
Bumping up this thread since another one seems to be starting over CI
Amogh
On 5/27/14 8:28 PM, "David Nalley" <da...@gnsa.us> wrote:
>On Tue, May 27, 2014 at 12:52 PM, Alex Huang <Al...@citrix.com>
>wrote:
>>> Like Chip, I am very concerned with this being dependent on a single
>>> company, even if its the company that employs me. It isn't
>>>sustainable, it
>>> excludes others from contributing, and makes the project less
>>>independent
>>> because it depends on a single company's infrastructure.
>>
>> Agreed there.
>>
>>>
>>> I'm also unclear on the answer to the question in the FAQ. The first
>>>time I
>>> read it, I got the impression that you were happy to bring it up on
>>>hardware
>>> at the ASF if the ASF wanted to own it. The second time I read it I
>>>wondered
>>> if you meant that Citrix was going to attempt to donate hardware.
>>>
>> Sorry if I did not make that clear. I meant the scripts/code that we
>>wrote are checked in publicly and we're willing to help set it up if ASF
>>provided the hardware. I have not approach Citrix on donating the
>>actual hardware. Although I can approach them if it speeds up the
>>adoption process.
>>
>>> Finally - what do you think you need from ASF infra to make this
>>>happen?
>>>
>>
>> It's currently about 10 servers with two networks. One network is
>>static with IPMI to PXE boot the machines. The other network is the
>>actual data network that CloudStack uses. That's actually just enough
>>for XenServer and KVM. In order to accommodate for HyperV, Bare Metal,
>>LXC, (which we do not have any test cases in the automation suits
>>currently) we will need even more machines. We might be able to use
>>nested virtualization for the hypervisors to maintain server count at
>>ten or a little more than ten but we haven't explore that yet.
>>
>> The CI process is up and running on those machines but because we
>>didn't have CI running on master before, automation tests that were
>>passing for 4.3 are now broken again on 4.4. and master. I think Sudha
>>already reported on the list that QA is busy trying to fix all the
>>automation tests to bring CI on 4.4-forward and master back to 100% pass
>>rate. Unfortunately, it's been delaying our effort to put this out in
>>the public and let the community try this themselves.
>>
>> --Alex
>>
>
>So the board just approved a 3 month budget, but the new board will
>have to take up the remainder of the FY budget shortly after coming
>into office. Perhaps worth coming up with an estimate of what this
>will cost/need and getting it to president@ before that new budget is
>taken up.
>
>--David
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by David Nalley <da...@gnsa.us>.
On Tue, May 27, 2014 at 12:52 PM, Alex Huang <Al...@citrix.com> wrote:
>> Like Chip, I am very concerned with this being dependent on a single
>> company, even if its the company that employs me. It isn't sustainable, it
>> excludes others from contributing, and makes the project less independent
>> because it depends on a single company's infrastructure.
>
> Agreed there.
>
>>
>> I'm also unclear on the answer to the question in the FAQ. The first time I
>> read it, I got the impression that you were happy to bring it up on hardware
>> at the ASF if the ASF wanted to own it. The second time I read it I wondered
>> if you meant that Citrix was going to attempt to donate hardware.
>>
> Sorry if I did not make that clear. I meant the scripts/code that we wrote are checked in publicly and we're willing to help set it up if ASF provided the hardware. I have not approach Citrix on donating the actual hardware. Although I can approach them if it speeds up the adoption process.
>
>> Finally - what do you think you need from ASF infra to make this happen?
>>
>
> It's currently about 10 servers with two networks. One network is static with IPMI to PXE boot the machines. The other network is the actual data network that CloudStack uses. That's actually just enough for XenServer and KVM. In order to accommodate for HyperV, Bare Metal, LXC, (which we do not have any test cases in the automation suits currently) we will need even more machines. We might be able to use nested virtualization for the hypervisors to maintain server count at ten or a little more than ten but we haven't explore that yet.
>
> The CI process is up and running on those machines but because we didn't have CI running on master before, automation tests that were passing for 4.3 are now broken again on 4.4. and master. I think Sudha already reported on the list that QA is busy trying to fix all the automation tests to bring CI on 4.4-forward and master back to 100% pass rate. Unfortunately, it's been delaying our effort to put this out in the public and let the community try this themselves.
>
> --Alex
>
So the board just approved a 3 month budget, but the new board will
have to take up the remainder of the FY budget shortly after coming
into office. Perhaps worth coming up with an estimate of what this
will cost/need and getting it to president@ before that new budget is
taken up.
--David
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
> Like Chip, I am very concerned with this being dependent on a single
> company, even if its the company that employs me. It isn't sustainable, it
> excludes others from contributing, and makes the project less independent
> because it depends on a single company's infrastructure.
Agreed there.
>
> I'm also unclear on the answer to the question in the FAQ. The first time I
> read it, I got the impression that you were happy to bring it up on hardware
> at the ASF if the ASF wanted to own it. The second time I read it I wondered
> if you meant that Citrix was going to attempt to donate hardware.
>
Sorry if I did not make that clear. I meant the scripts/code that we wrote are checked in publicly and we're willing to help set it up if ASF provided the hardware. I have not approach Citrix on donating the actual hardware. Although I can approach them if it speeds up the adoption process.
> Finally - what do you think you need from ASF infra to make this happen?
>
It's currently about 10 servers with two networks. One network is static with IPMI to PXE boot the machines. The other network is the actual data network that CloudStack uses. That's actually just enough for XenServer and KVM. In order to accommodate for HyperV, Bare Metal, LXC, (which we do not have any test cases in the automation suits currently) we will need even more machines. We might be able to use nested virtualization for the hypervisors to maintain server count at ten or a little more than ten but we haven't explore that yet.
The CI process is up and running on those machines but because we didn't have CI running on master before, automation tests that were passing for 4.3 are now broken again on 4.4. and master. I think Sudha already reported on the list that QA is busy trying to fix all the automation tests to bring CI on 4.4-forward and master back to 100% pass rate. Unfortunately, it's been delaying our effort to put this out in the public and let the community try this themselves.
--Alex
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by David Nalley <da...@gnsa.us>.
> @Chip and @Hugo
>
>> Correct, and my statement stands. I'm -1 on any policy within the project
>> that enforces the use of a single company's resources if they are only
>> controlled by that company. Let's see how we can move this to the ASF (and
>> tweak / tune based on a better understanding of usage) before considering
>> it a project "policy".
>
> I think that's fair. And if anything in this proposal that I worried about, this is it. The problem for me is we haven't gotten there for ASF infra in two years and I'm worried that given the problems we're seeing with 4.4, we will lose momentum if we simply wait. I'm perfectly fine with exploring ways to find a middle ground. For example, if Citrix is to open the infrastructure to community (I would need to find some way to convince Citrix), is that acceptable? Opening it to the public would still mean the on-going cost is provided by Citrix so it's not truly independent of Citrix. I'm open to any suggestions to move us forward faster here.
>
Like Chip, I am very concerned with this being dependent on a single
company, even if its the company that employs me. It isn't
sustainable, it excludes others from contributing, and makes the
project less independent because it depends on a single company's
infrastructure.
I'm also unclear on the answer to the question in the FAQ. The first
time I read it, I got the impression that you were happy to bring it
up on hardware at the ASF if the ASF wanted to own it. The second time
I read it I wondered if you meant that Citrix was going to attempt to
donate hardware.
Finally - what do you think you need from ASF infra to make this happen?
--David
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Alex Huang <Al...@citrix.com>.
Sorry for the late replies on this folks. I've been away for my $dayjob at Citrix. Let me try to see if I can make clear what I see as objections/questions/understandings and address them one by one. If I got them wrong, let me know and please clarify what you're asking.
@Sebastien
> Why don't Citrix developers show us how they would do it in the open ? Right now it's all hidden.
I take this to indicate that there's a misunderstanding that this is a Citrix policy that the Citrix developers are following currently in secret and we're trying to ask the community to follow as well.
The fact is Citrix has no such policy currently and I'm trying to get Citrix management to agree to this checkin policy because of how poorly the previous releases have been going as well. I proposed this mainly because I see that because the project is so broad in scope that without a CI system to give people a fundamental comfort that they didn't break anything, it will be very difficult for the project to move forward. If Citrix developers were doing this before, we wouldn't have seen some of the reverts and large number of RCs in current and previous releases.
@Chip and @Hugo
> Correct, and my statement stands. I'm -1 on any policy within the project
> that enforces the use of a single company's resources if they are only
> controlled by that company. Let's see how we can move this to the ASF (and
> tweak / tune based on a better understanding of usage) before considering
> it a project "policy".
I think that's fair. And if anything in this proposal that I worried about, this is it. The problem for me is we haven't gotten there for ASF infra in two years and I'm worried that given the problems we're seeing with 4.4, we will lose momentum if we simply wait. I'm perfectly fine with exploring ways to find a middle ground. For example, if Citrix is to open the infrastructure to community (I would need to find some way to convince Citrix), is that acceptable? Opening it to the public would still mean the on-going cost is provided by Citrix so it's not truly independent of Citrix. I'm open to any suggestions to move us forward faster here.
@Hugo
> Anyway part of being a committer is the responsibility to make a correct decision when and how to commit to the central code base, this includes deciding when running automation tests is appropriate. You know i’m in favor of quality controls and i am a strong proponent of testing before committing, but each committer has his own responsibility in this and has to show he/she takes this seriously.
While I do trust the committers to be responsible (which is what I gather from the above quote), the problem is that the project is too broad and deep. Just being responsible is not enough to ensure that quality is not adversely affected. As a committer, I rather give up a bit of my liberties than to see the community produce poor quality releases. That's why I made this proposal.
BTW, I especially apologize for the late reply on this email. I think your email illustrated Sebastien and Chip's points as well but it came in to my inbox really late due to the mailing list problems so it was buried under a bunch of other emails.
Let me know what we can do, particularly on the how we can make this an independent process quickly.
--Alex
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Chip Childers <ch...@apache.org>.
On Mon, May 26, 2014 at 3:12 PM, Animesh Chaturvedi
<an...@citrix.com> wrote:
>
>>
>> I'll also state for the record, that I will -1 any decision that leads us to
>> depend on a single company's infrastructure as a project level policy. That
>> said, having it available from Citrix for use is a great thing for that company
>> to donate! It's going to be difficult to balance our development needs with
>> our need to remain vendor neutral / independent.
>>
>> -chip
> [Animesh] Chip Alex has called this out in Q&A section of [1]
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
>
Correct, and my statement stands. I'm -1 on any policy within the
project that enforces the use of a single company's resources if they
are only controlled by that company. Let's see how we can move this
to the ASF (and tweak / tune based on a better understanding of usage)
before considering it a project "policy".
-chip
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Animesh Chaturvedi <an...@citrix.com>.
>
> I'll also state for the record, that I will -1 any decision that leads us to
> depend on a single company's infrastructure as a project level policy. That
> said, having it available from Citrix for use is a great thing for that company
> to donate! It's going to be difficult to balance our development needs with
> our need to remain vendor neutral / independent.
>
> -chip
[Animesh] Chip Alex has called this out in Q&A section of [1]
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Chip Childers <ch...@apache.org>.
On Wed, May 21, 2014 at 2:30 AM, sebgoa <ru...@gmail.com> wrote:
> Why don't Citrix developers show us how they would do it in the open ? Right now it's all hidden.
+1 - I like some of the concepts of this proposal, but I have some
concerns as well.
I'll also state for the record, that I will -1 any decision that leads
us to depend on a single company's infrastructure as a project level
policy. That said, having it available from Citrix for use is a great
thing for that company to donate! It's going to be difficult to
balance our development needs with our need to remain vendor neutral /
independent.
-chip
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: sebgoa [mailto:runseb@gmail.com]
> Sent: Tuesday, May 20, 2014 11:30 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
>
> On May 21, 2014, at 12:40 AM, Animesh Chaturvedi
> <an...@citrix.com> wrote:
>
> > Hugo without putting certain rules we are not going to get out of the
> situation. I think it may seem draconian but stating "No merges are allowed
> without a successful run of the automation"
>
> Let's be specific. Are we talking about merges of features or every commit.
[Animesh] Alex can correct me if my interpretation is incorrect. But the idea is to protect release and master branch. Folks can continue to checkin into their own branch but before merge into the release branches the tests should pass for every commit
>
> A successful run of all integration tests with the simulator can take 45
> minutes maybe more. This means that everyone will have to wait for this to
> complete for even a typo.
>
> > is beneficial for everyone. It forces focus on automation and helps mitigate
> regression. If we run into challenges in implementing those rules like toolset
> not ready or infra then we would have to fix those.
> >
>
> Why don't Citrix developers show us how they would do it in the open ?
> Right now it's all hidden.
[Animesh] Certainly that is the intent and the proposal calls it out, we will have teething trouble initially but hopefully we will grow over it quickly
>
> > Animesh
> >
> >> -----Original Message-----
> >> From: Trippie [mailto:trippie@gmail.com] On Behalf Of Hugo Trippaers
> >> Sent: Wednesday, May 14, 2014 12:32 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: [PROPOSAL] Using continuous integration to maintain our
> >> code quality...
> >>
> >> Hey Alex,
> >>
> >> Nice job on getting this all done and working on some guidelines to
> >> improve quality of the overal guidelines. We discussed a lot of these
> >> things at the last CCC and i'm happy to see them here.
> >>
> >> I do have some feedback on the policy though.
> >>
> >> Specifically with the line stating "No merges are allowed without a
> >> successful run of the automation". You stated yourself that the infra
> >> running the automation is being run by Citrix. Introducing this
> >> statement links our policy to Citrix in such a way that we can't
> >> commit if the citrix supplied Infra is not availalbe. That doesn't
> >> sound like a good thing. Anyway part of being a committer is the
> >> responsibility to make a correct decision when and how to commit to
> >> the central code base, this includes deciding when running automation
> >> tests is appropriate. You know i'm in favor of quality controls and i
> >> am a strong proponent of testing before committing, but each
> >> committer has his own responsibility in this and has to show he/she takes
> this seriously.
> >>
> >> The JIRA process is pretty good and will certainly help with keeping
> >> track of what is going on, but is not mandatory in my opinion. Also
> >> arranging for a review is not a responsibility of the developer of a
> >> piece of code. If that is necessary it is the community that fails,
> >> the really responsibility to do code reviews is with the committers,
> >> "Each committer has a responsibility to monitor the changes made for
> potential issues, both coding and legal."
> >> (http://www.apache.org/dev/new-committers-guide.html).
> >>
> >> I'm a firm believer of providing tooling and support to help make
> >> "doing the right thing" as easy as possible. In my opinion the focus
> >> should be on making sure this tooling is as easy to use as possible
> >> so committers and contributors will want to use this, because it
> >> helps them instead of forcing them to use it by policy.
> >>
> >> Cheers,
> >>
> >> Hugo
> >>
> >>
> >> On 7 mei 2014, at 02:03, Alex Huang <Al...@citrix.com> wrote:
> >>
> >>> Hi All,
> >>>
> >>> This is something I brought up a long time ago but really didn't
> >>> have the
> >> resources to get it all up and running until now. Throughout the
> >> past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and
> >> others have been chipping away at it. At this point, we finally pull
> >> together a continuous integration setup that we can use to make sure
> >> that CloudStack master and the currently release branch are always
> >> stable. This is getting pretty close to be completed and we like to
> >> share it with the community in hopes that we can reduce/eliminate that
> problems we've seen with our recent releases.
> >> Currently, the physical hardware are hosted by Citrix but we'll be
> >> more than willing to donate the work to infra when that's all settled.
> >>>
> >>> This does require effort from the community to make a change in
> >>> their
> >> development process. These steps are detailed at [1]. I like to get
> >> feedback on what everyone think about this.
> >>>
> >>> What have we done:
> >>> - We replaced a large selection of the BVT tests to run with the
> >>> simulator
> >> instead of actual hardware. This shortens the duration of each BVT run.
> >> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> >> minutes.
> >>> - We will run the new BVT on master and the current release branch
> >>> on a
> >> continuous basis.
> >>> - Developers can use Jenkins to ask BVT to be run on their branch so
> >>> they
> >> can know it won't break the continuous integration before they merge
> >> to master and the current release branch.
> >>>
> >>> Please have a read and let me know what you think.
> >>>
> >>> --Alex
> >>>
> >>> [1]
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Pr
> >> o
> >> cess
> >
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by sebgoa <ru...@gmail.com>.
On May 21, 2014, at 12:40 AM, Animesh Chaturvedi <an...@citrix.com> wrote:
> Hugo without putting certain rules we are not going to get out of the situation. I think it may seem draconian but stating "No merges are allowed without a successful run of the automation"
Let's be specific. Are we talking about merges of features or every commit.
A successful run of all integration tests with the simulator can take 45 minutes maybe more. This means that everyone will have to wait for this to complete for even a typo.
> is beneficial for everyone. It forces focus on automation and helps mitigate regression. If we run into challenges in implementing those rules like toolset not ready or infra then we would have to fix those.
>
Why don't Citrix developers show us how they would do it in the open ? Right now it's all hidden.
> Animesh
>
>> -----Original Message-----
>> From: Trippie [mailto:trippie@gmail.com] On Behalf Of Hugo Trippaers
>> Sent: Wednesday, May 14, 2014 12:32 AM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
>> quality...
>>
>> Hey Alex,
>>
>> Nice job on getting this all done and working on some guidelines to improve
>> quality of the overal guidelines. We discussed a lot of these things at the last
>> CCC and i'm happy to see them here.
>>
>> I do have some feedback on the policy though.
>>
>> Specifically with the line stating "No merges are allowed without a successful
>> run of the automation". You stated yourself that the infra running the
>> automation is being run by Citrix. Introducing this statement links our policy
>> to Citrix in such a way that we can't commit if the citrix supplied Infra is not
>> availalbe. That doesn't sound like a good thing. Anyway part of being a
>> committer is the responsibility to make a correct decision when and how to
>> commit to the central code base, this includes deciding when running
>> automation tests is appropriate. You know i'm in favor of quality controls
>> and i am a strong proponent of testing before committing, but each
>> committer has his own responsibility in this and has to show he/she takes
>> this seriously.
>>
>> The JIRA process is pretty good and will certainly help with keeping track of
>> what is going on, but is not mandatory in my opinion. Also arranging for a
>> review is not a responsibility of the developer of a piece of code. If that is
>> necessary it is the community that fails, the really responsibility to do code
>> reviews is with the committers, "Each committer has a responsibility to
>> monitor the changes made for potential issues, both coding and legal."
>> (http://www.apache.org/dev/new-committers-guide.html).
>>
>> I'm a firm believer of providing tooling and support to help make "doing the
>> right thing" as easy as possible. In my opinion the focus should be on making
>> sure this tooling is as easy to use as possible so committers and contributors
>> will want to use this, because it helps them instead of forcing them to use it
>> by policy.
>>
>> Cheers,
>>
>> Hugo
>>
>>
>> On 7 mei 2014, at 02:03, Alex Huang <Al...@citrix.com> wrote:
>>
>>> Hi All,
>>>
>>> This is something I brought up a long time ago but really didn't have the
>> resources to get it all up and running until now. Throughout the past year,
>> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
>> chipping away at it. At this point, we finally pull together a continuous
>> integration setup that we can use to make sure that CloudStack master and
>> the currently release branch are always stable. This is getting pretty close to
>> be completed and we like to share it with the community in hopes that we
>> can reduce/eliminate that problems we've seen with our recent releases.
>> Currently, the physical hardware are hosted by Citrix but we'll be more than
>> willing to donate the work to infra when that's all settled.
>>>
>>> This does require effort from the community to make a change in their
>> development process. These steps are detailed at [1]. I like to get feedback
>> on what everyone think about this.
>>>
>>> What have we done:
>>> - We replaced a large selection of the BVT tests to run with the simulator
>> instead of actual hardware. This shortens the duration of each BVT run.
>> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
>> minutes.
>>> - We will run the new BVT on master and the current release branch on a
>> continuous basis.
>>> - Developers can use Jenkins to ask BVT to be run on their branch so they
>> can know it won't break the continuous integration before they merge to
>> master and the current release branch.
>>>
>>> Please have a read and let me know what you think.
>>>
>>> --Alex
>>>
>>> [1]
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Pro
>> cess
>
RE: [PROPOSAL] Using continuous integration to maintain our code
quality...
Posted by Animesh Chaturvedi <an...@citrix.com>.
Hugo without putting certain rules we are not going to get out of the situation. I think it may seem draconian but stating "No merges are allowed without a successful run of the automation" is beneficial for everyone. It forces focus on automation and helps mitigate regression. If we run into challenges in implementing those rules like toolset not ready or infra then we would have to fix those.
Animesh
> -----Original Message-----
> From: Trippie [mailto:trippie@gmail.com] On Behalf Of Hugo Trippaers
> Sent: Wednesday, May 14, 2014 12:32 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Using continuous integration to maintain our code
> quality...
>
> Hey Alex,
>
> Nice job on getting this all done and working on some guidelines to improve
> quality of the overal guidelines. We discussed a lot of these things at the last
> CCC and i'm happy to see them here.
>
> I do have some feedback on the policy though.
>
> Specifically with the line stating "No merges are allowed without a successful
> run of the automation". You stated yourself that the infra running the
> automation is being run by Citrix. Introducing this statement links our policy
> to Citrix in such a way that we can't commit if the citrix supplied Infra is not
> availalbe. That doesn't sound like a good thing. Anyway part of being a
> committer is the responsibility to make a correct decision when and how to
> commit to the central code base, this includes deciding when running
> automation tests is appropriate. You know i'm in favor of quality controls
> and i am a strong proponent of testing before committing, but each
> committer has his own responsibility in this and has to show he/she takes
> this seriously.
>
> The JIRA process is pretty good and will certainly help with keeping track of
> what is going on, but is not mandatory in my opinion. Also arranging for a
> review is not a responsibility of the developer of a piece of code. If that is
> necessary it is the community that fails, the really responsibility to do code
> reviews is with the committers, "Each committer has a responsibility to
> monitor the changes made for potential issues, both coding and legal."
> (http://www.apache.org/dev/new-committers-guide.html).
>
> I'm a firm believer of providing tooling and support to help make "doing the
> right thing" as easy as possible. In my opinion the focus should be on making
> sure this tooling is as easy to use as possible so committers and contributors
> will want to use this, because it helps them instead of forcing them to use it
> by policy.
>
> Cheers,
>
> Hugo
>
>
> On 7 mei 2014, at 02:03, Alex Huang <Al...@citrix.com> wrote:
>
> > Hi All,
> >
> > This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now. Throughout the past year,
> Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been
> chipping away at it. At this point, we finally pull together a continuous
> integration setup that we can use to make sure that CloudStack master and
> the currently release branch are always stable. This is getting pretty close to
> be completed and we like to share it with the community in hopes that we
> can reduce/eliminate that problems we've seen with our recent releases.
> Currently, the physical hardware are hosted by Citrix but we'll be more than
> willing to donate the work to infra when that's all settled.
> >
> > This does require effort from the community to make a change in their
> development process. These steps are detailed at [1]. I like to get feedback
> on what everyone think about this.
> >
> > What have we done:
> > - We replaced a large selection of the BVT tests to run with the simulator
> instead of actual hardware. This shortens the duration of each BVT run.
> Today, a BVT that runs tests for XenServer and KVM completes in 30-40
> minutes.
> > - We will run the new BVT on master and the current release branch on a
> continuous basis.
> > - Developers can use Jenkins to ask BVT to be run on their branch so they
> can know it won't break the continuous integration before they merge to
> master and the current release branch.
> >
> > Please have a read and let me know what you think.
> >
> > --Alex
> >
> > [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Pro
> cess
Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Posted by Hugo Trippaers <hu...@apache.org>.
Hey Alex,
Nice job on getting this all done and working on some guidelines to improve quality of the overal guidelines. We discussed a lot of these things at the last CCC and i’m happy to see them here.
I do have some feedback on the policy though.
Specifically with the line stating “No merges are allowed without a successful run of the automation”. You stated yourself that the infra running the automation is being run by Citrix. Introducing this statement links our policy to Citrix in such a way that we can’t commit if the citrix supplied Infra is not availalbe. That doesn’t sound like a good thing. Anyway part of being a committer is the responsibility to make a correct decision when and how to commit to the central code base, this includes deciding when running automation tests is appropriate. You know i’m in favor of quality controls and i am a strong proponent of testing before committing, but each committer has his own responsibility in this and has to show he/she takes this seriously.
The JIRA process is pretty good and will certainly help with keeping track of what is going on, but is not mandatory in my opinion. Also arranging for a review is not a responsibility of the developer of a piece of code. If that is necessary it is the community that fails, the really responsibility to do code reviews is with the committers, "Each committer has a responsibility to monitor the changes made for potential issues, both coding and legal.” (http://www.apache.org/dev/new-committers-guide.html).
I’m a firm believer of providing tooling and support to help make “doing the right thing” as easy as possible. In my opinion the focus should be on making sure this tooling is as easy to use as possible so committers and contributors will want to use this, because it helps them instead of forcing them to use it by policy.
Cheers,
Hugo
On 7 mei 2014, at 02:03, Alex Huang <Al...@citrix.com> wrote:
> Hi All,
>
> This is something I brought up a long time ago but really didn't have the resources to get it all up and running until now. Throughout the past year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have been chipping away at it. At this point, we finally pull together a continuous integration setup that we can use to make sure that CloudStack master and the currently release branch are always stable. This is getting pretty close to be completed and we like to share it with the community in hopes that we can reduce/eliminate that problems we've seen with our recent releases. Currently, the physical hardware are hosted by Citrix but we'll be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their development process. These steps are detailed at [1]. I like to get feedback on what everyone think about this.
>
> What have we done:
> - We replaced a large selection of the BVT tests to run with the simulator instead of actual hardware. This shortens the duration of each BVT run. Today, a BVT that runs tests for XenServer and KVM completes in 30-40 minutes.
> - We will run the new BVT on master and the current release branch on a continuous basis.
> - Developers can use Jenkins to ask BVT to be run on their branch so they can know it won't break the continuous integration before they merge to master and the current release branch.
>
> Please have a read and let me know what you think.
>
> --Alex
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+Process