You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Sebastien Goasguen <ru...@gmail.com> on 2016/01/27 21:18:55 UTC

[PROPOSAL] Freeze everything until we get CI

Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.

-Sebastien

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Bharat Kumar <bh...@citrix.com>.

Sure Remi,

Once I am done consolidating the scripts i will put them in a separate repo on github.

Thanks,
Bharat.


> On 30-Jan-2016, at 5:14 PM, Remi Bergsma <RB...@schubergphilis.com> wrote:
> 
> Please put it in a separate repo. There’s way too much stuff in the cloudstack repo already, IMHO and we should be splitting out, not adding more :-)
> 
> Regards,
> Remi
> 
> 
> 
> 
> 
> On 29/01/16 08:22, "Erik Weber" <te...@gmail.com> wrote:
> 
>> I'd love to see this in the cloudstack repository.
>> Others might have an easier time getting access to hardware, and could use
>> it to help test releases/PR
>> 
>> 
>> Erik
>> 
>> Den fredag 29. januar 2016 skrev Bharat Kumar <bh...@citrix.com>
>> følgende:
>> 
>>> yes, we would be sharing it with the community and get this running in the
>>> ACS infra.
>>> Currently it can create a cloudstack test bed, runs tests and email the
>>> results.
>>> 
>>> Here are some details on how this works and what is needed to set this up.
>>> 
>>>  *   we use jenkins, cobbler, puppet and marvin to create cloudstack
>>> setup.
>>>  *   jenkins triggers the test runs, collects the test results and mails
>>> them.
>>>  *   cobbler is use to image the hosts and create Management server.
>>>  *   The management server is a VM and each time a test run is triggered
>>> we pull the latest code, build (dev setup) the MS and run it.
>>>  *   Need IPMI enabled servers to uses and Hosts in cloudstack setup.
>>> Cobbler installs the required OS on these hosts.
>>>  *   We use a XenServer to create management server VMs.
>>> 
>>> The resources required to set this up.
>>> 
>>>  *   We need two servers to host the VMs used in CI, one XenServer to
>>> host the Cloustack management servers and at least 3 IPMI enabled servers
>>> per cloudstack setup to run the BVTs.
>>>  *   some set of IPs (public and private IPs) and vlans.
>>> 
>>> Once we have the resources in ACS infra we can start setting this up. But
>>> some work needs to
>>> be done to integrate this with the github to test and post the results in
>>> the PRs instead of mailing them.
>>> 
>>> I think the best way to share it will be by implementing this in the ACS
>>> infra. Once we do this every one can pitch in, replicate and further
>>> contribute to this.
>>> 
>>> Meanwhile i will commit the scripts to set this up and keep this going.
>>> 
>>> Thanks,
>>> Bharat.
>>> 
>>> 
>>> On 28-Jan-2016, at 7:37 PM, Erik Weber <terbolous@gmail.com <javascript:;>
>>> <mailto:terbolous@gmail.com <javascript:;>>> wrote:
>>> 
>>> Why not share it as is, then the community could help improving this,
>>> rather than this being a single company effort?
>>> 
>>> --
>>> Erik
>>> 
>>> On Thu, Jan 28, 2016 at 1:49 PM, Bharat Kumar <bharat.kumar@citrix.com
>>> <javascript:;><mailto:bharat.kumar@citrix.com <javascript:;>>>
>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I agree that we need to have a CI to deal with the large volume of PRs.
>>> The current travis CI is not good enough as it runs only simulator tests.
>>> We identified this issue and came up with a effective CI for automating
>>> test runs for a each PR. This is already functional, with few github
>>> integration aspects pending. We are internally stabilizing it before
>>> sharing it.
>>> 
>>> We have been in touch with David Nalley ( CC’ed )  in making this
>>> operational for entire community using ACS infra.
>>> 
>>> 
>>> For your reference, here is the FS I have shared with the community
>>> earlier and also in this thread before, your feedback is welcome.
>>> (
>>> 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration
>>> ).
>>> 
>>> Thanks,
>>> Bharat.
>>> 
>>> 
>>> 
>>> 
>>> On 28-Jan-2016, at 4:26 PM, Rohit Yadav <rohit.yadav@shapeblue.com
>>> <javascript:;><mailto:
>>> rohit.yadav@shapeblue.com <javascript:;>>> wrote:
>>> 
>>> All,
>>> 
>>> I’m sorry to get to have the PRs merged without adhering to the strict
>>> testing requirements. While I think PRs were alright and it did not break
>>> anything, the way it was merged made people uncomfortable that there is
>>> some sort of haste in doing this fast which there is none.
>>> 
>>> I’ll not repeat this and hope you understand that I never had any hidden
>>> agenda but to simply help people with some PRs.
>>> 
>>> Regards.
>>> 
>>> On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <runseb@gmail.com
>>> <javascript:;><mailto:
>>> runseb@gmail.com <javascript:;>>> wrote:
>>> 
>>> Hi Folks,
>>> 
>>> My proposal to freeze until we get CI was indeed due to seeing Rohit’s
>>> commit but was by no means a personal attack or judgment.
>>> 
>>> We have lots of PR pending (as mentioned before by Remi) and we need
>>> people to help review and test.
>>> So thanks to Rohit.
>>> 
>>> My only concerns were two fold:
>>> 
>>> 1- We need  to keep to adhere to our release principles:
>>> 
>>> 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>>> 
>>> Hence I replied to some PR asking if they needed to be merged directly in
>>> master or not and wondered about the release branches.
>>> 
>>> With so many releases in flight it is not yet clear to me where we start
>>> to apply a PR ?
>>> 
>>> 2- We need to keep testing and post results of tests.
>>> 
>>> Currently it is manual and but there has been a strong guarantee in the
>>> last releases that the PR where not going to break things.
>>> While I agree that some PR are small and *should* not break things,
>>> history has shown that even small unrelated things *somehow* can affect the
>>> behavior of cloudstack.
>>> 
>>> So I proposed a freeze because:
>>> 
>>> - Remi stepped down as RM and we don’t have an official RM yet.
>>> - The code has reached a solid state and we don’t want to do anything that
>>> changes that
>>> - We have a proposal for LTS on the floor
>>> - We still don’t have CI.
>>> 
>>> So my standpoint is that we focused in the last 6 months on getting our
>>> release principles right (pending LTS principles), code has stabilized and
>>> we can release. Awesome.
>>> 
>>> Now is probably a good time to concentrate our limited resources on
>>> figuring out automated CI.
>>> 
>>> - For instance as far as I know Travis is bonkers…(reports green but does
>>> not do anything)
>>> - And with citrix stepping out, we need to take control of the jenkins
>>> slaves (some of which are on AWS and still paid by Citrix…)
>>> 
>>> My email while triggered by seeing Rohit’s commits, was not a judgement or
>>> critic of his actions, so let’s not get into a personal argument here.
>>> 
>>> -Sebastien
>>> 
>>> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <rohit.yadav@shapeblue.com
>>> <javascript:;>>
>>> wrote:
>>> 
>>> So, since some have directly (over IM etc) or indirectly have thrown
>>> allegations on me since I merged most of the PRs.
>>> Here is a list of those 12 PRs and answers on why they were merged on
>>> case-by-case basis.
>>> Please keep any further replies technical and to the specific PR, please
>>> point out and revert if needed:
>>> 
>>> 1. https://github.com/apache/cloudstack/pull/1288
>>> 
>>> Enough LGTMs, JS related change and fix tested with UI screenshot from
>>> Remi. I personally looked at the diff and therefore then merged.
>>> 
>>> 2. https://github.com/apache/cloudstack/pull/1274/files
>>> 
>>> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can
>>> cheat here and given Travis/Jenkins passed I merged it.
>>> 
>>> 3. https://github.com/apache/cloudstack/pull/1261/files
>>> 
>>> Enough LGTMs, the diff only removed unused variable leading to change in
>>> the constructor definition. Explicit integration tests are not necessary as
>>> it’d simply dead-code removal and as the simulator smoke tests passed with
>>> Travis/Jenkins passed so I merged it.
>>> 
>>> 4. https://github.com/apache/cloudstack/pull/1048
>>> 
>>> Enough LGTMs. This change is related to a marvin test itself where it adds
>>> 2 new test methods — so no need to run regression integration test. The
>>> integration test result of the marvin test was shared in the comment. PR
>>> merged on this basis.
>>> 
>>> 5. https://github.com/apache/cloudstack/pull/1044
>>> 
>>> Enough LGTMs and regression tests results (shared as attachments by Daan,
>>> in case someone missed), so merged.
>>> 
>>> 6. https://github.com/apache/cloudstack/pull/969
>>> 
>>> Enough LGTMs and regression tests results (shared as attachments by Daan,
>>> in case someone missed), so merged.
>>> 
>>> 7. https://github.com/apache/cloudstack/pull/855
>>> 
>>> Enough LGTMs and regression tests results by Remi, so merged.
>>> 
>>> 8. https://github.com/apache/cloudstack/pull/831
>>> 
>>> Enough LGTMs and only text changes in API doc-string so merged given
>>> Travis/Jenkins passed.
>>> 
>>> 9. https://github.com/apache/cloudstack/pull/830/files
>>> 
>>> Enough LGTMs and NPE fixes, so no explicit integration tests required
>>> given Travis/Jenkins passed.
>>> 
>>> 10. https://github.com/apache/cloudstack/pull/1256/files
>>> 
>>> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged
>>> this. I’m aware of this codebase.
>>> 
>>> 11. https://github.com/apache/cloudstack/pull/1240
>>> 
>>> Enough LGTMs, the changes would require manual tests wrt usage server etc
>>> as well as confirmed in comments. I had seen the regression test result (of
>>> the new/modified marvin test wrt of the feature) so merged. The regression
>>> test suite does not include this among other tests.
>>> 
>>> 12. https://github.com/apache/cloudstack/pull/1289
>>> 
>>> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it
>>> and the findbugs mvn job result was shared to confirm that the fix works
>>> now. This was not merged by me.
>>> 
>>> Regards.
>>> 
>>> 
>>> 
>>> Rohit Yadav
>>> Software Architect   ,       ShapeBlue
>>> d:    | s: +44 203 603 0540   |      m:      +91 8826230892
>>> e:   rohit.yadav@shapeblue.com <javascript:;> | t:   |      w:
>>> www.shapeblue.com
>>> a:   53 Chandos Place, Covent Garden London WC2N 4HS UK
>>> 
>>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>>> Services India LLP is a company incorporated in India and is operated under
>>> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
>>> company incorporated in Brasil and is operated under license from Shape
>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
>>> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
>>> a registered trademark.
>>> This email and any attachments to it may be confidential and are intended
>>> solely for the use of the individual to whom it is addressed. Any views or
>>> opinions expressed are solely those of the author and do not necessarily
>>> represent those of Shape Blue Ltd or related companies. If you are not the
>>> intended recipient of this email, you must neither take any action based
>>> upon its contents, nor copy or show it to anyone. Please contact the sender
>>> if you believe you have received this email in error.
>>> 
>>> 
>>> Find out more about ShapeBlue and our range of CloudStack related services:
>>> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
>>> CloudStack Consulting | CloudStack Software Engineering
>>> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
>>> 
>>> 
>>> Regards.
>>> 
>>> Find out more about ShapeBlue and our range of CloudStack related services:
>>> IaaS Cloud Design & Build<
>>> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>>> IaaS deployment framework<http://shapeblue.com/csforge/>
>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>>> CloudStack Software Engineering<
>>> http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure Support<
>>> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>>> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>> 
>>> 
>>> 
>>>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Remi Bergsma <RB...@schubergphilis.com>.

Please put it in a separate repo. There’s way too much stuff in the cloudstack repo already, IMHO and we should be splitting out, not adding more :-)

Regards,
Remi





On 29/01/16 08:22, "Erik Weber" <te...@gmail.com> wrote:

>I'd love to see this in the cloudstack repository.
>Others might have an easier time getting access to hardware, and could use
>it to help test releases/PR
>
>
>Erik
>
>Den fredag 29. januar 2016 skrev Bharat Kumar <bh...@citrix.com>
>følgende:
>
>> yes, we would be sharing it with the community and get this running in the
>> ACS infra.
>> Currently it can create a cloudstack test bed, runs tests and email the
>> results.
>>
>> Here are some details on how this works and what is needed to set this up.
>>
>>   *   we use jenkins, cobbler, puppet and marvin to create cloudstack
>> setup.
>>   *   jenkins triggers the test runs, collects the test results and mails
>> them.
>>   *   cobbler is use to image the hosts and create Management server.
>>   *   The management server is a VM and each time a test run is triggered
>> we pull the latest code, build (dev setup) the MS and run it.
>>   *   Need IPMI enabled servers to uses and Hosts in cloudstack setup.
>> Cobbler installs the required OS on these hosts.
>>   *   We use a XenServer to create management server VMs.
>>
>> The resources required to set this up.
>>
>>   *   We need two servers to host the VMs used in CI, one XenServer to
>> host the Cloustack management servers and at least 3 IPMI enabled servers
>> per cloudstack setup to run the BVTs.
>>   *   some set of IPs (public and private IPs) and vlans.
>>
>> Once we have the resources in ACS infra we can start setting this up. But
>> some work needs to
>> be done to integrate this with the github to test and post the results in
>> the PRs instead of mailing them.
>>
>> I think the best way to share it will be by implementing this in the ACS
>> infra. Once we do this every one can pitch in, replicate and further
>> contribute to this.
>>
>> Meanwhile i will commit the scripts to set this up and keep this going.
>>
>> Thanks,
>> Bharat.
>>
>>
>> On 28-Jan-2016, at 7:37 PM, Erik Weber <terbolous@gmail.com <javascript:;>
>> <mailto:terbolous@gmail.com <javascript:;>>> wrote:
>>
>> Why not share it as is, then the community could help improving this,
>> rather than this being a single company effort?
>>
>> --
>> Erik
>>
>> On Thu, Jan 28, 2016 at 1:49 PM, Bharat Kumar <bharat.kumar@citrix.com
>> <javascript:;><mailto:bharat.kumar@citrix.com <javascript:;>>>
>> wrote:
>>
>> Hi All,
>>
>> I agree that we need to have a CI to deal with the large volume of PRs.
>> The current travis CI is not good enough as it runs only simulator tests.
>> We identified this issue and came up with a effective CI for automating
>> test runs for a each PR. This is already functional, with few github
>> integration aspects pending. We are internally stabilizing it before
>> sharing it.
>>
>> We have been in touch with David Nalley ( CC’ed )  in making this
>> operational for entire community using ACS infra.
>>
>>
>> For your reference, here is the FS I have shared with the community
>> earlier and also in this thread before, your feedback is welcome.
>> (
>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration
>> ).
>>
>> Thanks,
>> Bharat.
>>
>>
>>
>>
>> On 28-Jan-2016, at 4:26 PM, Rohit Yadav <rohit.yadav@shapeblue.com
>> <javascript:;><mailto:
>> rohit.yadav@shapeblue.com <javascript:;>>> wrote:
>>
>> All,
>>
>> I’m sorry to get to have the PRs merged without adhering to the strict
>> testing requirements. While I think PRs were alright and it did not break
>> anything, the way it was merged made people uncomfortable that there is
>> some sort of haste in doing this fast which there is none.
>>
>> I’ll not repeat this and hope you understand that I never had any hidden
>> agenda but to simply help people with some PRs.
>>
>> Regards.
>>
>> On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <runseb@gmail.com
>> <javascript:;><mailto:
>> runseb@gmail.com <javascript:;>>> wrote:
>>
>> Hi Folks,
>>
>> My proposal to freeze until we get CI was indeed due to seeing Rohit’s
>> commit but was by no means a personal attack or judgment.
>>
>> We have lots of PR pending (as mentioned before by Remi) and we need
>> people to help review and test.
>> So thanks to Rohit.
>>
>> My only concerns were two fold:
>>
>> 1- We need  to keep to adhere to our release principles:
>>
>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>>
>> Hence I replied to some PR asking if they needed to be merged directly in
>> master or not and wondered about the release branches.
>>
>> With so many releases in flight it is not yet clear to me where we start
>> to apply a PR ?
>>
>> 2- We need to keep testing and post results of tests.
>>
>> Currently it is manual and but there has been a strong guarantee in the
>> last releases that the PR where not going to break things.
>> While I agree that some PR are small and *should* not break things,
>> history has shown that even small unrelated things *somehow* can affect the
>> behavior of cloudstack.
>>
>> So I proposed a freeze because:
>>
>> - Remi stepped down as RM and we don’t have an official RM yet.
>> - The code has reached a solid state and we don’t want to do anything that
>> changes that
>> - We have a proposal for LTS on the floor
>> - We still don’t have CI.
>>
>> So my standpoint is that we focused in the last 6 months on getting our
>> release principles right (pending LTS principles), code has stabilized and
>> we can release. Awesome.
>>
>> Now is probably a good time to concentrate our limited resources on
>> figuring out automated CI.
>>
>> - For instance as far as I know Travis is bonkers…(reports green but does
>> not do anything)
>> - And with citrix stepping out, we need to take control of the jenkins
>> slaves (some of which are on AWS and still paid by Citrix…)
>>
>> My email while triggered by seeing Rohit’s commits, was not a judgement or
>> critic of his actions, so let’s not get into a personal argument here.
>>
>> -Sebastien
>>
>> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <rohit.yadav@shapeblue.com
>> <javascript:;>>
>> wrote:
>>
>> So, since some have directly (over IM etc) or indirectly have thrown
>> allegations on me since I merged most of the PRs.
>> Here is a list of those 12 PRs and answers on why they were merged on
>> case-by-case basis.
>> Please keep any further replies technical and to the specific PR, please
>> point out and revert if needed:
>>
>> 1. https://github.com/apache/cloudstack/pull/1288
>>
>> Enough LGTMs, JS related change and fix tested with UI screenshot from
>> Remi. I personally looked at the diff and therefore then merged.
>>
>> 2. https://github.com/apache/cloudstack/pull/1274/files
>>
>> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can
>> cheat here and given Travis/Jenkins passed I merged it.
>>
>> 3. https://github.com/apache/cloudstack/pull/1261/files
>>
>> Enough LGTMs, the diff only removed unused variable leading to change in
>> the constructor definition. Explicit integration tests are not necessary as
>> it’d simply dead-code removal and as the simulator smoke tests passed with
>> Travis/Jenkins passed so I merged it.
>>
>> 4. https://github.com/apache/cloudstack/pull/1048
>>
>> Enough LGTMs. This change is related to a marvin test itself where it adds
>> 2 new test methods — so no need to run regression integration test. The
>> integration test result of the marvin test was shared in the comment. PR
>> merged on this basis.
>>
>> 5. https://github.com/apache/cloudstack/pull/1044
>>
>> Enough LGTMs and regression tests results (shared as attachments by Daan,
>> in case someone missed), so merged.
>>
>> 6. https://github.com/apache/cloudstack/pull/969
>>
>> Enough LGTMs and regression tests results (shared as attachments by Daan,
>> in case someone missed), so merged.
>>
>> 7. https://github.com/apache/cloudstack/pull/855
>>
>> Enough LGTMs and regression tests results by Remi, so merged.
>>
>> 8. https://github.com/apache/cloudstack/pull/831
>>
>> Enough LGTMs and only text changes in API doc-string so merged given
>> Travis/Jenkins passed.
>>
>> 9. https://github.com/apache/cloudstack/pull/830/files
>>
>> Enough LGTMs and NPE fixes, so no explicit integration tests required
>> given Travis/Jenkins passed.
>>
>> 10. https://github.com/apache/cloudstack/pull/1256/files
>>
>> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged
>> this. I’m aware of this codebase.
>>
>> 11. https://github.com/apache/cloudstack/pull/1240
>>
>> Enough LGTMs, the changes would require manual tests wrt usage server etc
>> as well as confirmed in comments. I had seen the regression test result (of
>> the new/modified marvin test wrt of the feature) so merged. The regression
>> test suite does not include this among other tests.
>>
>> 12. https://github.com/apache/cloudstack/pull/1289
>>
>> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it
>> and the findbugs mvn job result was shared to confirm that the fix works
>> now. This was not merged by me.
>>
>> Regards.
>>
>>
>>
>> Rohit Yadav
>> Software Architect   ,       ShapeBlue
>> d:    | s: +44 203 603 0540   |      m:      +91 8826230892
>> e:   rohit.yadav@shapeblue.com <javascript:;> | t:   |      w:
>> www.shapeblue.com
>> a:   53 Chandos Place, Covent Garden London WC2N 4HS UK
>>
>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>> Services India LLP is a company incorporated in India and is operated under
>> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
>> company incorporated in Brasil and is operated under license from Shape
>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
>> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
>> a registered trademark.
>> This email and any attachments to it may be confidential and are intended
>> solely for the use of the individual to whom it is addressed. Any views or
>> opinions expressed are solely those of the author and do not necessarily
>> represent those of Shape Blue Ltd or related companies. If you are not the
>> intended recipient of this email, you must neither take any action based
>> upon its contents, nor copy or show it to anyone. Please contact the sender
>> if you believe you have received this email in error.
>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
>> CloudStack Consulting | CloudStack Software Engineering
>> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
>>
>>
>> Regards.
>>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build<
>> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>> IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>> CloudStack Software Engineering<
>> http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<
>> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>
>>
>>
>>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Erik Weber <te...@gmail.com>.

I'd love to see this in the cloudstack repository.
Others might have an easier time getting access to hardware, and could use
it to help test releases/PR


Erik

Den fredag 29. januar 2016 skrev Bharat Kumar <bh...@citrix.com>
følgende:

> yes, we would be sharing it with the community and get this running in the
> ACS infra.
> Currently it can create a cloudstack test bed, runs tests and email the
> results.
>
> Here are some details on how this works and what is needed to set this up.
>
>   *   we use jenkins, cobbler, puppet and marvin to create cloudstack
> setup.
>   *   jenkins triggers the test runs, collects the test results and mails
> them.
>   *   cobbler is use to image the hosts and create Management server.
>   *   The management server is a VM and each time a test run is triggered
> we pull the latest code, build (dev setup) the MS and run it.
>   *   Need IPMI enabled servers to uses and Hosts in cloudstack setup.
> Cobbler installs the required OS on these hosts.
>   *   We use a XenServer to create management server VMs.
>
> The resources required to set this up.
>
>   *   We need two servers to host the VMs used in CI, one XenServer to
> host the Cloustack management servers and at least 3 IPMI enabled servers
> per cloudstack setup to run the BVTs.
>   *   some set of IPs (public and private IPs) and vlans.
>
> Once we have the resources in ACS infra we can start setting this up. But
> some work needs to
> be done to integrate this with the github to test and post the results in
> the PRs instead of mailing them.
>
> I think the best way to share it will be by implementing this in the ACS
> infra. Once we do this every one can pitch in, replicate and further
> contribute to this.
>
> Meanwhile i will commit the scripts to set this up and keep this going.
>
> Thanks,
> Bharat.
>
>
> On 28-Jan-2016, at 7:37 PM, Erik Weber <terbolous@gmail.com <javascript:;>
> <mailto:terbolous@gmail.com <javascript:;>>> wrote:
>
> Why not share it as is, then the community could help improving this,
> rather than this being a single company effort?
>
> --
> Erik
>
> On Thu, Jan 28, 2016 at 1:49 PM, Bharat Kumar <bharat.kumar@citrix.com
> <javascript:;><mailto:bharat.kumar@citrix.com <javascript:;>>>
> wrote:
>
> Hi All,
>
> I agree that we need to have a CI to deal with the large volume of PRs.
> The current travis CI is not good enough as it runs only simulator tests.
> We identified this issue and came up with a effective CI for automating
> test runs for a each PR. This is already functional, with few github
> integration aspects pending. We are internally stabilizing it before
> sharing it.
>
> We have been in touch with David Nalley ( CC’ed )  in making this
> operational for entire community using ACS infra.
>
>
> For your reference, here is the FS I have shared with the community
> earlier and also in this thread before, your feedback is welcome.
> (
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration
> ).
>
> Thanks,
> Bharat.
>
>
>
>
> On 28-Jan-2016, at 4:26 PM, Rohit Yadav <rohit.yadav@shapeblue.com
> <javascript:;><mailto:
> rohit.yadav@shapeblue.com <javascript:;>>> wrote:
>
> All,
>
> I’m sorry to get to have the PRs merged without adhering to the strict
> testing requirements. While I think PRs were alright and it did not break
> anything, the way it was merged made people uncomfortable that there is
> some sort of haste in doing this fast which there is none.
>
> I’ll not repeat this and hope you understand that I never had any hidden
> agenda but to simply help people with some PRs.
>
> Regards.
>
> On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <runseb@gmail.com
> <javascript:;><mailto:
> runseb@gmail.com <javascript:;>>> wrote:
>
> Hi Folks,
>
> My proposal to freeze until we get CI was indeed due to seeing Rohit’s
> commit but was by no means a personal attack or judgment.
>
> We have lots of PR pending (as mentioned before by Remi) and we need
> people to help review and test.
> So thanks to Rohit.
>
> My only concerns were two fold:
>
> 1- We need  to keep to adhere to our release principles:
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>
> Hence I replied to some PR asking if they needed to be merged directly in
> master or not and wondered about the release branches.
>
> With so many releases in flight it is not yet clear to me where we start
> to apply a PR ?
>
> 2- We need to keep testing and post results of tests.
>
> Currently it is manual and but there has been a strong guarantee in the
> last releases that the PR where not going to break things.
> While I agree that some PR are small and *should* not break things,
> history has shown that even small unrelated things *somehow* can affect the
> behavior of cloudstack.
>
> So I proposed a freeze because:
>
> - Remi stepped down as RM and we don’t have an official RM yet.
> - The code has reached a solid state and we don’t want to do anything that
> changes that
> - We have a proposal for LTS on the floor
> - We still don’t have CI.
>
> So my standpoint is that we focused in the last 6 months on getting our
> release principles right (pending LTS principles), code has stabilized and
> we can release. Awesome.
>
> Now is probably a good time to concentrate our limited resources on
> figuring out automated CI.
>
> - For instance as far as I know Travis is bonkers…(reports green but does
> not do anything)
> - And with citrix stepping out, we need to take control of the jenkins
> slaves (some of which are on AWS and still paid by Citrix…)
>
> My email while triggered by seeing Rohit’s commits, was not a judgement or
> critic of his actions, so let’s not get into a personal argument here.
>
> -Sebastien
>
> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <rohit.yadav@shapeblue.com
> <javascript:;>>
> wrote:
>
> So, since some have directly (over IM etc) or indirectly have thrown
> allegations on me since I merged most of the PRs.
> Here is a list of those 12 PRs and answers on why they were merged on
> case-by-case basis.
> Please keep any further replies technical and to the specific PR, please
> point out and revert if needed:
>
> 1. https://github.com/apache/cloudstack/pull/1288
>
> Enough LGTMs, JS related change and fix tested with UI screenshot from
> Remi. I personally looked at the diff and therefore then merged.
>
> 2. https://github.com/apache/cloudstack/pull/1274/files
>
> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can
> cheat here and given Travis/Jenkins passed I merged it.
>
> 3. https://github.com/apache/cloudstack/pull/1261/files
>
> Enough LGTMs, the diff only removed unused variable leading to change in
> the constructor definition. Explicit integration tests are not necessary as
> it’d simply dead-code removal and as the simulator smoke tests passed with
> Travis/Jenkins passed so I merged it.
>
> 4. https://github.com/apache/cloudstack/pull/1048
>
> Enough LGTMs. This change is related to a marvin test itself where it adds
> 2 new test methods — so no need to run regression integration test. The
> integration test result of the marvin test was shared in the comment. PR
> merged on this basis.
>
> 5. https://github.com/apache/cloudstack/pull/1044
>
> Enough LGTMs and regression tests results (shared as attachments by Daan,
> in case someone missed), so merged.
>
> 6. https://github.com/apache/cloudstack/pull/969
>
> Enough LGTMs and regression tests results (shared as attachments by Daan,
> in case someone missed), so merged.
>
> 7. https://github.com/apache/cloudstack/pull/855
>
> Enough LGTMs and regression tests results by Remi, so merged.
>
> 8. https://github.com/apache/cloudstack/pull/831
>
> Enough LGTMs and only text changes in API doc-string so merged given
> Travis/Jenkins passed.
>
> 9. https://github.com/apache/cloudstack/pull/830/files
>
> Enough LGTMs and NPE fixes, so no explicit integration tests required
> given Travis/Jenkins passed.
>
> 10. https://github.com/apache/cloudstack/pull/1256/files
>
> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged
> this. I’m aware of this codebase.
>
> 11. https://github.com/apache/cloudstack/pull/1240
>
> Enough LGTMs, the changes would require manual tests wrt usage server etc
> as well as confirmed in comments. I had seen the regression test result (of
> the new/modified marvin test wrt of the feature) so merged. The regression
> test suite does not include this among other tests.
>
> 12. https://github.com/apache/cloudstack/pull/1289
>
> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it
> and the findbugs mvn job result was shared to confirm that the fix works
> now. This was not merged by me.
>
> Regards.
>
>
>
> Rohit Yadav
> Software Architect   ,       ShapeBlue
> d:    | s: +44 203 603 0540   |      m:      +91 8826230892
> e:   rohit.yadav@shapeblue.com <javascript:;> | t:   |      w:
> www.shapeblue.com
> a:   53 Chandos Place, Covent Garden London WC2N 4HS UK
>
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
> CloudStack Consulting | CloudStack Software Engineering
> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
>
>
> Regards.
>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
>
>
>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Bharat Kumar <bh...@citrix.com>.

yes, we would be sharing it with the community and get this running in the ACS infra.
Currently it can create a cloudstack test bed, runs tests and email the results.

Here are some details on how this works and what is needed to set this up.

* we use jenkins, cobbler, puppet and marvin to create cloudstack setup.
* jenkins triggers the test runs, collects the test results and mails them.
* cobbler is use to image the hosts and create Management server.
* The management server is a VM and each time a test run is triggered we pull the latest code, build (dev setup) the MS and run it.
* Need IPMI enabled servers to uses and Hosts in cloudstack setup. Cobbler installs the required OS on these hosts.
* We use a XenServer to create management server VMs.

The resources required to set this up.

* We need two servers to host the VMs used in CI, one XenServer to host the Cloustack management servers and at least 3 IPMI enabled servers per cloudstack setup to run the BVTs.
* some set of IPs (public and private IPs) and vlans.

Once we have the resources in ACS infra we can start setting this up. But some work needs to
be done to integrate this with the github to test and post the results in the PRs instead of mailing them.

I think the best way to share it will be by implementing this in the ACS infra. Once we do this every one can pitch in, replicate and further contribute to this.

Meanwhile i will commit the scripts to set this up and keep this going.

Thanks,
Bharat.

On 28-Jan-2016, at 7:37 PM, Erik Weber <te...@gmail.com>> wrote:

Why not share it as is, then the community could help improving this,
rather than this being a single company effort?

--
Erik

On Thu, Jan 28, 2016 at 1:49 PM, Bharat Kumar <bh...@citrix.com>>
wrote:

Hi All,

I agree that we need to have a CI to deal with the large volume of PRs.
The current travis CI is not good enough as it runs only simulator tests.
We identified this issue and came up with a effective CI for automating
test runs for a each PR. This is already functional, with few github
integration aspects pending. We are internally stabilizing it before
sharing it.

We have been in touch with David Nalley ( CC’ed ) in making this
operational for entire community using ACS infra.

For your reference, here is the FS I have shared with the community
earlier and also in this thread before, your feedback is welcome.
(
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration
).

Thanks,
Bharat.

On 28-Jan-2016, at 4:26 PM, Rohit Yadav <rohit.yadav@shapeblue.com<mailto:
rohit.yadav@shapeblue.com>> wrote:

All,

I’m sorry to get to have the PRs merged without adhering to the strict
testing requirements. While I think PRs were alright and it did not break
anything, the way it was merged made people uncomfortable that there is
some sort of haste in doing this fast which there is none.

I’ll not repeat this and hope you understand that I never had any hidden
agenda but to simply help people with some PRs.

Regards.

On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <runseb@gmail.com<mailto:
runseb@gmail.com>> wrote:

Hi Folks,

My proposal to freeze until we get CI was indeed due to seeing Rohit’s
commit but was by no means a personal attack or judgment.

We have lots of PR pending (as mentioned before by Remi) and we need
people to help review and test.
So thanks to Rohit.

My only concerns were two fold:

1- We need to keep to adhere to our release principles:

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up

Hence I replied to some PR asking if they needed to be merged directly in
master or not and wondered about the release branches.

With so many releases in flight it is not yet clear to me where we start
to apply a PR ?

2- We need to keep testing and post results of tests.

Currently it is manual and but there has been a strong guarantee in the
last releases that the PR where not going to break things.
While I agree that some PR are small and *should* not break things,
history has shown that even small unrelated things *somehow* can affect the
behavior of cloudstack.

So I proposed a freeze because:

- Remi stepped down as RM and we don’t have an official RM yet.
- The code has reached a solid state and we don’t want to do anything that
changes that
- We have a proposal for LTS on the floor
- We still don’t have CI.

So my standpoint is that we focused in the last 6 months on getting our
release principles right (pending LTS principles), code has stabilized and
we can release. Awesome.

Now is probably a good time to concentrate our limited resources on
figuring out automated CI.

- For instance as far as I know Travis is bonkers…(reports green but does
not do anything)
- And with citrix stepping out, we need to take control of the jenkins
slaves (some of which are on AWS and still paid by Citrix…)

My email while triggered by seeing Rohit’s commits, was not a judgement or
critic of his actions, so let’s not get into a personal argument here.

-Sebastien

On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com>
wrote:

So, since some have directly (over IM etc) or indirectly have thrown
allegations on me since I merged most of the PRs.
Here is a list of those 12 PRs and answers on why they were merged on
case-by-case basis.
Please keep any further replies technical and to the specific PR, please
point out and revert if needed:

1. https://github.com/apache/cloudstack/pull/1288

Enough LGTMs, JS related change and fix tested with UI screenshot from
Remi. I personally looked at the diff and therefore then merged.

2. https://github.com/apache/cloudstack/pull/1274/files

Enough LGTMs, a simple NPE fix one-liner. I personally thought we can
cheat here and given Travis/Jenkins passed I merged it.

3. https://github.com/apache/cloudstack/pull/1261/files

Enough LGTMs, the diff only removed unused variable leading to change in
the constructor definition. Explicit integration tests are not necessary as
it’d simply dead-code removal and as the simulator smoke tests passed with
Travis/Jenkins passed so I merged it.

4. https://github.com/apache/cloudstack/pull/1048

Enough LGTMs. This change is related to a marvin test itself where it adds
2 new test methods — so no need to run regression integration test. The
integration test result of the marvin test was shared in the comment. PR
merged on this basis.

5. https://github.com/apache/cloudstack/pull/1044

Enough LGTMs and regression tests results (shared as attachments by Daan,
in case someone missed), so merged.

6. https://github.com/apache/cloudstack/pull/969

Enough LGTMs and regression tests results (shared as attachments by Daan,
in case someone missed), so merged.

7. https://github.com/apache/cloudstack/pull/855

Enough LGTMs and regression tests results by Remi, so merged.

8. https://github.com/apache/cloudstack/pull/831

Enough LGTMs and only text changes in API doc-string so merged given
Travis/Jenkins passed.

9. https://github.com/apache/cloudstack/pull/830/files

Enough LGTMs and NPE fixes, so no explicit integration tests required
given Travis/Jenkins passed.

10. https://github.com/apache/cloudstack/pull/1256/files

Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged
this. I’m aware of this codebase.

11. https://github.com/apache/cloudstack/pull/1240

Enough LGTMs, the changes would require manual tests wrt usage server etc
as well as confirmed in comments. I had seen the regression test result (of
the new/modified marvin test wrt of the feature) so merged. The regression
test suite does not include this among other tests.

12. https://github.com/apache/cloudstack/pull/1289

Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it
and the findbugs mvn job result was shared to confirm that the fix works
now. This was not merged by me.

Regards.

Rohit Yadav
Software Architect , ShapeBlue
d: | s: +44 203 603 0540 | m: +91 8826230892
e: rohit.yadav@shapeblue.com | t: | w: www.shapeblue.com
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK

Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
Services India LLP is a company incorporated in India and is operated under
license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
company incorporated in Brasil and is operated under license from Shape
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
a registered trademark.
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Shape Blue Ltd or related companies. If you are not the
intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender
if you believe you have received this email in error.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
CloudStack Consulting | CloudStack Software Engineering
CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<
http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
CloudStack Software Engineering<
http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<
http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Erik Weber <te...@gmail.com>.

Why not share it as is, then the community could help improving this,
rather than this being a single company effort?

-- 
Erik

On Thu, Jan 28, 2016 at 1:49 PM, Bharat Kumar <bh...@citrix.com>
wrote:

> Hi All,
>
> I agree that we need to have a CI to deal with the large volume of PRs.
> The current travis CI is not good enough as it runs only simulator tests.
> We identified this issue and came up with a effective CI for automating
> test runs for a each PR. This is already functional, with few github
> integration aspects pending. We are internally stabilizing it before
> sharing it.
>
> We have been in touch with David Nalley ( CC’ed )  in making this
> operational for entire community using ACS infra.
>
>
> For your reference, here is the FS I have shared with the community
> earlier and also in this thread before, your feedback is welcome.
> (
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration
> ).
>
> Thanks,
> Bharat.
>
>
>
>
> On 28-Jan-2016, at 4:26 PM, Rohit Yadav <rohit.yadav@shapeblue.com<mailto:
> rohit.yadav@shapeblue.com>> wrote:
>
> All,
>
> I’m sorry to get to have the PRs merged without adhering to the strict
> testing requirements. While I think PRs were alright and it did not break
> anything, the way it was merged made people uncomfortable that there is
> some sort of haste in doing this fast which there is none.
>
> I’ll not repeat this and hope you understand that I never had any hidden
> agenda but to simply help people with some PRs.
>
> Regards.
>
> On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <runseb@gmail.com<mailto:
> runseb@gmail.com>> wrote:
>
> Hi Folks,
>
> My proposal to freeze until we get CI was indeed due to seeing Rohit’s
> commit but was by no means a personal attack or judgment.
>
> We have lots of PR pending (as mentioned before by Remi) and we need
> people to help review and test.
> So thanks to Rohit.
>
> My only concerns were two fold:
>
> 1- We need  to keep to adhere to our release principles:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>
> Hence I replied to some PR asking if they needed to be merged directly in
> master or not and wondered about the release branches.
>
> With so many releases in flight it is not yet clear to me where we start
> to apply a PR ?
>
> 2- We need to keep testing and post results of tests.
>
> Currently it is manual and but there has been a strong guarantee in the
> last releases that the PR where not going to break things.
> While I agree that some PR are small and *should* not break things,
> history has shown that even small unrelated things *somehow* can affect the
> behavior of cloudstack.
>
> So I proposed a freeze because:
>
> - Remi stepped down as RM and we don’t have an official RM yet.
> - The code has reached a solid state and we don’t want to do anything that
> changes that
> - We have a proposal for LTS on the floor
> - We still don’t have CI.
>
> So my standpoint is that we focused in the last 6 months on getting our
> release principles right (pending LTS principles), code has stabilized and
> we can release. Awesome.
>
> Now is probably a good time to concentrate our limited resources on
> figuring out automated CI.
>
> - For instance as far as I know Travis is bonkers…(reports green but does
> not do anything)
> - And with citrix stepping out, we need to take control of the jenkins
> slaves (some of which are on AWS and still paid by Citrix…)
>
> My email while triggered by seeing Rohit’s commits, was not a judgement or
> critic of his actions, so let’s not get into a personal argument here.
>
> -Sebastien
>
> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> So, since some have directly (over IM etc) or indirectly have thrown
> allegations on me since I merged most of the PRs.
> Here is a list of those 12 PRs and answers on why they were merged on
> case-by-case basis.
> Please keep any further replies technical and to the specific PR, please
> point out and revert if needed:
>
> 1. https://github.com/apache/cloudstack/pull/1288
>
> Enough LGTMs, JS related change and fix tested with UI screenshot from
> Remi. I personally looked at the diff and therefore then merged.
>
> 2. https://github.com/apache/cloudstack/pull/1274/files
>
> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can
> cheat here and given Travis/Jenkins passed I merged it.
>
> 3. https://github.com/apache/cloudstack/pull/1261/files
>
> Enough LGTMs, the diff only removed unused variable leading to change in
> the constructor definition. Explicit integration tests are not necessary as
> it’d simply dead-code removal and as the simulator smoke tests passed with
> Travis/Jenkins passed so I merged it.
>
> 4. https://github.com/apache/cloudstack/pull/1048
>
> Enough LGTMs. This change is related to a marvin test itself where it adds
> 2 new test methods — so no need to run regression integration test. The
> integration test result of the marvin test was shared in the comment. PR
> merged on this basis.
>
> 5. https://github.com/apache/cloudstack/pull/1044
>
> Enough LGTMs and regression tests results (shared as attachments by Daan,
> in case someone missed), so merged.
>
> 6. https://github.com/apache/cloudstack/pull/969
>
> Enough LGTMs and regression tests results (shared as attachments by Daan,
> in case someone missed), so merged.
>
> 7. https://github.com/apache/cloudstack/pull/855
>
> Enough LGTMs and regression tests results by Remi, so merged.
>
> 8. https://github.com/apache/cloudstack/pull/831
>
> Enough LGTMs and only text changes in API doc-string so merged given
> Travis/Jenkins passed.
>
> 9. https://github.com/apache/cloudstack/pull/830/files
>
> Enough LGTMs and NPE fixes, so no explicit integration tests required
> given Travis/Jenkins passed.
>
> 10. https://github.com/apache/cloudstack/pull/1256/files
>
> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged
> this. I’m aware of this codebase.
>
> 11. https://github.com/apache/cloudstack/pull/1240
>
> Enough LGTMs, the changes would require manual tests wrt usage server etc
> as well as confirmed in comments. I had seen the regression test result (of
> the new/modified marvin test wrt of the feature) so merged. The regression
> test suite does not include this among other tests.
>
> 12. https://github.com/apache/cloudstack/pull/1289
>
> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it
> and the findbugs mvn job result was shared to confirm that the fix works
> now. This was not merged by me.
>
> Regards.
>
>
>
> Rohit Yadav
> Software Architect   ,       ShapeBlue
> d:    | s: +44 203 603 0540   |      m:      +91 8826230892
> e:   rohit.yadav@shapeblue.com | t:   |      w:      www.shapeblue.com
> a:   53 Chandos Place, Covent Garden London WC2N 4HS UK
>
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
> CloudStack Consulting | CloudStack Software Engineering
> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
>
>
> Regards.
>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
>

RE: [PROPOSAL] Freeze everything until we get CI

Posted by Paul Angus <pa...@shapeblue.com>.

Hi Kumar,

That’s great. I've been working along similar lines, I'd love to collaborate with you on this.

Are you able to share an estimated timescale with us? And are you thinking of sharing the code and construction details of how to build the environment(s) for the community and users to build or are you looking to deploy the CI capability on ASF infra for the community to consume?



Paul Angus
VP Technology   ,       ShapeBlue


t:      @cloudyangus<te...@cloudyangus>

e:      paul.angus@shapeblue.com<ma...@shapeblue.com>        |      w:      www.shapeblue.com<http://www.shapeblue.com>





-----Original Message-----
From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
Sent: 28 January 2016 21:50
To: <de...@cloudstack.apache.org> <de...@cloudstack.apache.org>
Cc: David Nalley <da...@gnsa.us>
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi All,

I agree that we need to have a CI to deal with the large volume of PRs. The current travis CI is not good enough as it runs only simulator tests.
We identified this issue and came up with a effective CI for automating test runs for a each PR. This is already functional, with few github integration aspects pending. We are internally stabilizing it before sharing it.

We have been in touch with David Nalley ( CC’ed ) in making this operational for entire community using ACS infra.


For your reference, here is the FS I have shared with the community earlier and also in this thread before, your feedback is welcome.
(https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration).

Thanks,
Bharat.




On 28-Jan-2016, at 4:26 PM, Rohit Yadav <ro...@shapeblue.com>> wrote:

All,

I’m sorry to get to have the PRs merged without adhering to the strict testing requirements. While I think PRs were alright and it did not break anything, the way it was merged made people uncomfortable that there is some sort of haste in doing this fast which there is none.

I’ll not repeat this and hope you understand that I never had any hidden agenda but to simply help people with some PRs.

Regards.

On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <ru...@gmail.com>> wrote:

Hi Folks,

My proposal to freeze until we get CI was indeed due to seeing Rohit’s commit but was by no means a personal attack or judgment.

We have lots of PR pending (as mentioned before by Remi) and we need people to help review and test.
So thanks to Rohit.

My only concerns were two fold:

1- We need to keep to adhere to our release principles:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up

Hence I replied to some PR asking if they needed to be merged directly in master or not and wondered about the release branches.

With so many releases in flight it is not yet clear to me where we start to apply a PR ?

2- We need to keep testing and post results of tests.

Currently it is manual and but there has been a strong guarantee in the last releases that the PR where not going to break things.
While I agree that some PR are small and *should* not break things, history has shown that even small unrelated things *somehow* can affect the behavior of cloudstack.

So I proposed a freeze because:

- Remi stepped down as RM and we don’t have an official RM yet.
- The code has reached a solid state and we don’t want to do anything that changes that
- We have a proposal for LTS on the floor
- We still don’t have CI.

So my standpoint is that we focused in the last 6 months on getting our release principles right (pending LTS principles), code has stabilized and we can release. Awesome.

Now is probably a good time to concentrate our limited resources on figuring out automated CI.

- For instance as far as I know Travis is bonkers…(reports green but does not do anything)
- And with citrix stepping out, we need to take control of the jenkins slaves (some of which are on AWS and still paid by Citrix…)

My email while triggered by seeing Rohit’s commits, was not a judgement or critic of his actions, so let’s not get into a personal argument here.

-Sebastien

On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com> wrote:

So, since some have directly (over IM etc) or indirectly have thrown allegations on me since I merged most of the PRs.
Here is a list of those 12 PRs and answers on why they were merged on case-by-case basis.
Please keep any further replies technical and to the specific PR, please point out and revert if needed:

1. https://github.com/apache/cloudstack/pull/1288

Enough LGTMs, JS related change and fix tested with UI screenshot from Remi. I personally looked at the diff and therefore then merged.

2. https://github.com/apache/cloudstack/pull/1274/files

Enough LGTMs, a simple NPE fix one-liner. I personally thought we can cheat here and given Travis/Jenkins passed I merged it.

3. https://github.com/apache/cloudstack/pull/1261/files

Enough LGTMs, the diff only removed unused variable leading to change in the constructor definition. Explicit integration tests are not necessary as it’d simply dead-code removal and as the simulator smoke tests passed with Travis/Jenkins passed so I merged it.

4. https://github.com/apache/cloudstack/pull/1048

Enough LGTMs. This change is related to a marvin test itself where it adds 2 new test methods — so no need to run regression integration test. The integration test result of the marvin test was shared in the comment. PR merged on this basis.

5. https://github.com/apache/cloudstack/pull/1044

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

6. https://github.com/apache/cloudstack/pull/969

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

7. https://github.com/apache/cloudstack/pull/855

Enough LGTMs and regression tests results by Remi, so merged.

8. https://github.com/apache/cloudstack/pull/831

Enough LGTMs and only text changes in API doc-string so merged given Travis/Jenkins passed.

9. https://github.com/apache/cloudstack/pull/830/files

Enough LGTMs and NPE fixes, so no explicit integration tests required given Travis/Jenkins passed.

10. https://github.com/apache/cloudstack/pull/1256/files

Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged this. I’m aware of this codebase.

11. https://github.com/apache/cloudstack/pull/1240

Enough LGTMs, the changes would require manual tests wrt usage server etc as well as confirmed in comments. I had seen the regression test result (of the new/modified marvin test wrt of the feature) so merged. The regression test suite does not include this among other tests.

12. https://github.com/apache/cloudstack/pull/1289

Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it and the findbugs mvn job result was shared to confirm that the fix works now. This was not merged by me.

Regards.



Rohit Yadav
Software Architect , ShapeBlue
d: | s: +44 203 603 0540 | m: +91 8826230892
e: rohit.yadav@shapeblue.com | t: | w: www.shapeblue.com
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK

Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.


Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework CloudStack Consulting | CloudStack Software Engineering CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses


Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Bharat Kumar <bh...@citrix.com>.

Hi All,

I agree that we need to have a CI to deal with the large volume of PRs. The current travis CI is not good enough as it runs only simulator tests.
We identified this issue and came up with a effective CI for automating test runs for a each PR. This is already functional, with few github integration aspects pending. We are internally stabilizing it before sharing it.

We have been in touch with David Nalley ( CC’ed ) in making this operational for entire community using ACS infra.

For your reference, here is the FS I have shared with the community earlier and also in this thread before, your feedback is welcome.
(https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration).

Thanks,
Bharat.

On 28-Jan-2016, at 4:26 PM, Rohit Yadav <ro...@shapeblue.com>> wrote:

All,

I’m sorry to get to have the PRs merged without adhering to the strict testing requirements. While I think PRs were alright and it did not break anything, the way it was merged made people uncomfortable that there is some sort of haste in doing this fast which there is none.

I’ll not repeat this and hope you understand that I never had any hidden agenda but to simply help people with some PRs.

Regards.

On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <ru...@gmail.com>> wrote:

Hi Folks,

My proposal to freeze until we get CI was indeed due to seeing Rohit’s commit but was by no means a personal attack or judgment.

We have lots of PR pending (as mentioned before by Remi) and we need people to help review and test.
So thanks to Rohit.

My only concerns were two fold:

1- We need to keep to adhere to our release principles:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up

Hence I replied to some PR asking if they needed to be merged directly in master or not and wondered about the release branches.

With so many releases in flight it is not yet clear to me where we start to apply a PR ?

2- We need to keep testing and post results of tests.

Currently it is manual and but there has been a strong guarantee in the last releases that the PR where not going to break things.
While I agree that some PR are small and *should* not break things, history has shown that even small unrelated things *somehow* can affect the behavior of cloudstack.

So I proposed a freeze because:

- Remi stepped down as RM and we don’t have an official RM yet.
- The code has reached a solid state and we don’t want to do anything that changes that
- We have a proposal for LTS on the floor
- We still don’t have CI.

So my standpoint is that we focused in the last 6 months on getting our release principles right (pending LTS principles), code has stabilized and we can release. Awesome.

Now is probably a good time to concentrate our limited resources on figuring out automated CI.

- For instance as far as I know Travis is bonkers…(reports green but does not do anything)
- And with citrix stepping out, we need to take control of the jenkins slaves (some of which are on AWS and still paid by Citrix…)

My email while triggered by seeing Rohit’s commits, was not a judgement or critic of his actions, so let’s not get into a personal argument here.

-Sebastien

On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com> wrote:

So, since some have directly (over IM etc) or indirectly have thrown allegations on me since I merged most of the PRs.
Here is a list of those 12 PRs and answers on why they were merged on case-by-case basis.
Please keep any further replies technical and to the specific PR, please point out and revert if needed:

1. https://github.com/apache/cloudstack/pull/1288

Enough LGTMs, JS related change and fix tested with UI screenshot from Remi. I personally looked at the diff and therefore then merged.

2. https://github.com/apache/cloudstack/pull/1274/files

Enough LGTMs, a simple NPE fix one-liner. I personally thought we can cheat here and given Travis/Jenkins passed I merged it.

3. https://github.com/apache/cloudstack/pull/1261/files

Enough LGTMs, the diff only removed unused variable leading to change in the constructor definition. Explicit integration tests are not necessary as it’d simply dead-code removal and as the simulator smoke tests passed with Travis/Jenkins passed so I merged it.

4. https://github.com/apache/cloudstack/pull/1048

Enough LGTMs. This change is related to a marvin test itself where it adds 2 new test methods — so no need to run regression integration test. The integration test result of the marvin test was shared in the comment. PR merged on this basis.

5. https://github.com/apache/cloudstack/pull/1044

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

6. https://github.com/apache/cloudstack/pull/969

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

7. https://github.com/apache/cloudstack/pull/855

Enough LGTMs and regression tests results by Remi, so merged.

8. https://github.com/apache/cloudstack/pull/831

Enough LGTMs and only text changes in API doc-string so merged given Travis/Jenkins passed.

9. https://github.com/apache/cloudstack/pull/830/files

Enough LGTMs and NPE fixes, so no explicit integration tests required given Travis/Jenkins passed.

10. https://github.com/apache/cloudstack/pull/1256/files

Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged this. I’m aware of this codebase.

11. https://github.com/apache/cloudstack/pull/1240

Enough LGTMs, the changes would require manual tests wrt usage server etc as well as confirmed in comments. I had seen the regression test result (of the new/modified marvin test wrt of the feature) so merged. The regression test suite does not include this among other tests.

12. https://github.com/apache/cloudstack/pull/1289

Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it and the findbugs mvn job result was shared to confirm that the fix works now. This was not merged by me.

Regards.

Rohit Yadav
Software Architect , ShapeBlue
d: | s: +44 203 603 0540 | m: +91 8826230892
e: rohit.yadav@shapeblue.com | t: | w: www.shapeblue.com
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK

Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rohit Yadav <ro...@shapeblue.com>.

All,

I’m sorry to get to have the PRs merged without adhering to the strict testing requirements. While I think PRs were alright and it did not break anything, the way it was merged made people uncomfortable that there is some sort of haste in doing this fast which there is none.

I’ll not repeat this and hope you understand that I never had any hidden agenda but to simply help people with some PRs.

Regards.

> On 28-Jan-2016, at 11:36 AM, Sebastien Goasguen <ru...@gmail.com> wrote:
>
> Hi Folks,
>
> My proposal to freeze until we get CI was indeed due to seeing Rohit’s commit but was by no means a personal attack or judgment.
>
> We have lots of PR pending (as mentioned before by Remi) and we need people to help review and test.
> So thanks to Rohit.
>
> My only concerns were two fold:
>
> 1- We need  to keep to adhere to our release principles:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>
> Hence I replied to some PR asking if they needed to be merged directly in master or not and wondered about the release branches.
>
> With so many releases in flight it is not yet clear to me where we start to apply a PR ?
>
> 2- We need to keep testing and post results of tests.
>
> Currently it is manual and but there has been a strong guarantee in the last releases that the PR where not going to break things.
> While I agree that some PR are small and *should* not break things, history has shown that even small unrelated things *somehow* can affect the behavior of cloudstack.
>
> So I proposed a freeze because:
>
> - Remi stepped down as RM and we don’t have an official RM yet.
> - The code has reached a solid state and we don’t want to do anything that changes that
> - We have a proposal for LTS on the floor
> - We still don’t have CI.
>
> So my standpoint is that we focused in the last 6 months on getting our release principles right (pending LTS principles), code has stabilized and we can release. Awesome.
>
> Now is probably a good time to concentrate our limited resources on figuring out automated CI.
>
> - For instance as far as I know Travis is bonkers…(reports green but does not do anything)
> - And with citrix stepping out, we need to take control of the jenkins slaves (some of which are on AWS and still paid by Citrix…)
>
> My email while triggered by seeing Rohit’s commits, was not a judgement or critic of his actions, so let’s not get into a personal argument here.
>
> -Sebastien
>
>> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
>>
>> So, since some have directly (over IM etc) or indirectly have thrown allegations on me since I merged most of the PRs.
>> Here is a list of those 12 PRs and answers on why they were merged on case-by-case basis.
>> Please keep any further replies technical and to the specific PR, please point out and revert if needed:
>>
>> 1. https://github.com/apache/cloudstack/pull/1288
>>
>> Enough LGTMs, JS related change and fix tested with UI screenshot from Remi. I personally looked at the diff and therefore then merged.
>>
>> 2. https://github.com/apache/cloudstack/pull/1274/files
>>
>> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can cheat here and given Travis/Jenkins passed I merged it.
>>
>> 3. https://github.com/apache/cloudstack/pull/1261/files
>>
>> Enough LGTMs, the diff only removed unused variable leading to change in the constructor definition. Explicit integration tests are not necessary as it’d simply dead-code removal and as the simulator smoke tests passed with Travis/Jenkins passed so I merged it.
>>
>> 4. https://github.com/apache/cloudstack/pull/1048
>>
>> Enough LGTMs. This change is related to a marvin test itself where it adds 2 new test methods — so no need to run regression integration test. The integration test result of the marvin test was shared in the comment. PR merged on this basis.
>>
>> 5. https://github.com/apache/cloudstack/pull/1044
>>
>> Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.
>>
>> 6. https://github.com/apache/cloudstack/pull/969
>>
>> Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.
>>
>> 7. https://github.com/apache/cloudstack/pull/855
>>
>> Enough LGTMs and regression tests results by Remi, so merged.
>>
>> 8. https://github.com/apache/cloudstack/pull/831
>>
>> Enough LGTMs and only text changes in API doc-string so merged given Travis/Jenkins passed.
>>
>> 9. https://github.com/apache/cloudstack/pull/830/files
>>
>> Enough LGTMs and NPE fixes, so no explicit integration tests required given Travis/Jenkins passed.
>>
>> 10. https://github.com/apache/cloudstack/pull/1256/files
>>
>> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged this. I’m aware of this codebase.
>>
>> 11. https://github.com/apache/cloudstack/pull/1240
>>
>> Enough LGTMs, the changes would require manual tests wrt usage server etc as well as confirmed in comments. I had seen the regression test result (of the new/modified marvin test wrt of the feature) so merged. The regression test suite does not include this among other tests.
>>
>> 12. https://github.com/apache/cloudstack/pull/1289
>>
>> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it and the findbugs mvn job result was shared to confirm that the fix works now. This was not merged by me.
>>
>> Regards.
>>
>>
>>
>> Rohit Yadav
>> Software Architect   ,       ShapeBlue
>> d:    | s: +44 203 603 0540   |      m:      +91 8826230892
>> e:   rohit.yadav@shapeblue.com | t:   |      w:      www.shapeblue.com
>> a:   53 Chandos Place, Covent Garden London WC2N 4HS UK
>>
>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
>> CloudStack Consulting | CloudStack Software Engineering
>> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
>

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Sebastien Goasguen <ru...@gmail.com>.

Hi Folks,

My proposal to freeze until we get CI was indeed due to seeing Rohit’s commit but was by no means a personal attack or judgment.

We have lots of PR pending (as mentioned before by Remi) and we need people to help review and test.
So thanks to Rohit.

My only concerns were two fold:

1- We need  to keep to adhere to our release principles:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up

Hence I replied to some PR asking if they needed to be merged directly in master or not and wondered about the release branches.

With so many releases in flight it is not yet clear to me where we start to apply a PR ?

2- We need to keep testing and post results of tests.

Currently it is manual and but there has been a strong guarantee in the last releases that the PR where not going to break things.
While I agree that some PR are small and *should* not break things, history has shown that even small unrelated things *somehow* can affect the behavior of cloudstack.

So I proposed a freeze because:

- Remi stepped down as RM and we don’t have an official RM yet.
- The code has reached a solid state and we don’t want to do anything that changes that
- We have a proposal for LTS on the floor
- We still don’t have CI.

So my standpoint is that we focused in the last 6 months on getting our release principles right (pending LTS principles), code has stabilized and we can release. Awesome.

Now is probably a good time to concentrate our limited resources on figuring out automated CI.

- For instance as far as I know Travis is bonkers…(reports green but does not do anything)
- And with citrix stepping out, we need to take control of the jenkins slaves (some of which are on AWS and still paid by Citrix…)

My email while triggered by seeing Rohit’s commits, was not a judgement or critic of his actions, so let’s not get into a personal argument here.

-Sebastien

> On Jan 28, 2016, at 11:00 AM, Rohit Yadav <ro...@shapeblue.com> wrote:
> 
> So, since some have directly (over IM etc) or indirectly have thrown allegations on me since I merged most of the PRs.
> Here is a list of those 12 PRs and answers on why they were merged on case-by-case basis.
> Please keep any further replies technical and to the specific PR, please point out and revert if needed:
> 
> 1. https://github.com/apache/cloudstack/pull/1288
> 
> Enough LGTMs, JS related change and fix tested with UI screenshot from Remi. I personally looked at the diff and therefore then merged.
> 
> 2. https://github.com/apache/cloudstack/pull/1274/files
> 
> Enough LGTMs, a simple NPE fix one-liner. I personally thought we can cheat here and given Travis/Jenkins passed I merged it.
> 
> 3. https://github.com/apache/cloudstack/pull/1261/files
> 
> Enough LGTMs, the diff only removed unused variable leading to change in the constructor definition. Explicit integration tests are not necessary as it’d simply dead-code removal and as the simulator smoke tests passed with Travis/Jenkins passed so I merged it.
> 
> 4. https://github.com/apache/cloudstack/pull/1048
> 
> Enough LGTMs. This change is related to a marvin test itself where it adds 2 new test methods — so no need to run regression integration test. The integration test result of the marvin test was shared in the comment. PR merged on this basis.
> 
> 5. https://github.com/apache/cloudstack/pull/1044
> 
> Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.
> 
> 6. https://github.com/apache/cloudstack/pull/969
> 
> Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.
> 
> 7. https://github.com/apache/cloudstack/pull/855
> 
> Enough LGTMs and regression tests results by Remi, so merged.
> 
> 8. https://github.com/apache/cloudstack/pull/831
> 
> Enough LGTMs and only text changes in API doc-string so merged given Travis/Jenkins passed.
> 
> 9. https://github.com/apache/cloudstack/pull/830/files
> 
> Enough LGTMs and NPE fixes, so no explicit integration tests required given Travis/Jenkins passed.
> 
> 10. https://github.com/apache/cloudstack/pull/1256/files
> 
> Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged this. I’m aware of this codebase.
> 
> 11. https://github.com/apache/cloudstack/pull/1240
> 
> Enough LGTMs, the changes would require manual tests wrt usage server etc as well as confirmed in comments. I had seen the regression test result (of the new/modified marvin test wrt of the feature) so merged. The regression test suite does not include this among other tests.
> 
> 12. https://github.com/apache/cloudstack/pull/1289
> 
> Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it and the findbugs mvn job result was shared to confirm that the fix works now. This was not merged by me.
> 
> Regards.
> 
> 
> 	
> Rohit Yadav
> Software Architect	, 	ShapeBlue
> d: 	 | s: +44 203 603 0540	 | 	m: 	+91 8826230892
> e: 	rohit.yadav@shapeblue.com | t: 	 | 	w: 	www.shapeblue.com
> a: 	53 Chandos Place, Covent Garden London WC2N 4HS UK
> 
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
> 
> 
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build | CSForge – rapid IaaS deployment framework
> CloudStack Consulting | CloudStack Software Engineering
> CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rohit Yadav <ro...@shapeblue.com>.

So, since some have directly (over IM etc) or indirectly have thrown allegations on me since I merged most of the PRs.
Here is a list of those 12 PRs and answers on why they were merged on case-by-case basis.
Please keep any further replies technical and to the specific PR, please point out and revert if needed:

1. https://github.com/apache/cloudstack/pull/1288

Enough LGTMs, JS related change and fix tested with UI screenshot from Remi. I personally looked at the diff and therefore then merged.

2. https://github.com/apache/cloudstack/pull/1274/files

Enough LGTMs, a simple NPE fix one-liner. I personally thought we can cheat here and given Travis/Jenkins passed I merged it.

3. https://github.com/apache/cloudstack/pull/1261/files

Enough LGTMs, the diff only removed unused variable leading to change in the constructor definition. Explicit integration tests are not necessary as it’d simply dead-code removal and as the simulator smoke tests passed with Travis/Jenkins passed so I merged it.

4. https://github.com/apache/cloudstack/pull/1048

Enough LGTMs. This change is related to a marvin test itself where it adds 2 new test methods — so no need to run regression integration test. The integration test result of the marvin test was shared in the comment. PR merged on this basis.

5. https://github.com/apache/cloudstack/pull/1044

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

6. https://github.com/apache/cloudstack/pull/969

Enough LGTMs and regression tests results (shared as attachments by Daan, in case someone missed), so merged.

7. https://github.com/apache/cloudstack/pull/855

Enough LGTMs and regression tests results by Remi, so merged.

8. https://github.com/apache/cloudstack/pull/831

Enough LGTMs and only text changes in API doc-string so merged given Travis/Jenkins passed.

9. https://github.com/apache/cloudstack/pull/830/files

Enough LGTMs and NPE fixes, so no explicit integration tests required given Travis/Jenkins passed.

10. https://github.com/apache/cloudstack/pull/1256/files

Enough LGTMs and simple Java OOP fix with Travis/Jenkins passed so merged this. I’m aware of this codebase.

11. https://github.com/apache/cloudstack/pull/1240

Enough LGTMs, the changes would require manual tests wrt usage server etc as well as confirmed in comments. I had seen the regression test result (of the new/modified marvin test wrt of the feature) so merged. The regression test suite does not include this among other tests.

12. https://github.com/apache/cloudstack/pull/1289

Enough LGTMs, this was a findbugs related fix. Travis/Jenkins passed on it and the findbugs mvn job result was shared to confirm that the fix works now. This was not merged by me.

Regards.

[ShapeBlue]<http://www.shapeblue.com>
Rohit Yadav
Software Architect , ShapeBlue

d: | s: +44 203 603 0540<tel:|%20s:%20+44%20203%20603%200540> | m: +91 8826230892<tel:+91%208826230892>

e: rohit.yadav@shapeblue.com | t: <mailto:rohit.yadav@shapeblue.com%20|%20t:> | w: www.shapeblue.com<http://www.shapeblue.com>

a: 53 Chandos Place, Covent Garden London WC2N 4HS UK

[cid:imagec3b5d1.png@4f59c4e1.4a840a5d]

Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rohit Yadav <ro...@shapeblue.com>.


[ShapeBlue]<http://www.shapeblue.com>
Rohit Yadav
Software Architect      ,       ShapeBlue


d:       | s: +44 203 603 0540<tel:|%20s:%20+44%20203%20603%200540>      |      m:      +91 8826230892<tel:+91%208826230892>

e:      rohit.yadav@shapeblue.com | t: <mailto:rohit.yadav@shapeblue.com%20|%20t:>       |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:image62697c.png@c62a8c20.4d8db108]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




On 27-Jan-2016, at 9:29 PM, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:

>From the last 12 PRs that were merged, only 2 had integration tests executed against.

Feel free to revert them, if you must

I went through a list of pending PRs searching by lgtms, all of them has either text changes and/or integration tests passed/attached, and/or changes validated by travis/jenkins. According to me, they were validated that’s why they were merged.

These were PRs that had some discussions, at least one LGTM; master is not broken.

There is no rush, I had some time (which I don’t difficult these days) so helped review and merge them.

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Wilder Rodrigues <WR...@schubergphilis.com>.

+1 , Sebastien!

From the last 12 PRs that were merged, only 2 had integration tests executed against.

I don’t understand why the rush to merge things that were not properly tested. It already looks like a broken master and 4 votes to get 1 release through every 9 months. I should not make such a comment, but I can’t help it after seeing that Remi sent an email yesterday at 21h43 CET and today, 21h28, we are already facing such a situation.

Would be nice to get to know why it has to be like this, even if it doesn’t make sense.

Cheers,
Wilder

> On 27 Jan 2016, at 21:18, Sebastien Goasguen <ru...@gmail.com> wrote:
> 
> Folks,
> 
> How about we freeze our repo entirely until we get proper CI in place.
> 
> Seems to me all the hard work from Remi and co could be lost if we start committing again.
> 
> Now Travis is not running, Jenkins fails all the time and nobody cares…
> 
> So how about we figure out CI now ? and not do anything else.
> 
> -Sebastien

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Daan Hoogland <da...@gmail.com>.

On Wed, Jan 27, 2016 at 9:18 PM, Sebastien Goasguen <ru...@gmail.com>
wrote:

> Now Travis is not running, Jenkins fails all the time and nobody cares…
>

Sebastien, Jenkins succeeds more then 50% of the time and I do care.
Please report problems with jenkins.



-- 
Daan

RE: [PROPOSAL] Freeze everything until we get CI

Posted by Paul Angus <pa...@shapeblue.com>.

Thanks Remi,

That would give us a working 'good' baseline to start from - I've found tests that have hardcoded local IPs in template URLs and allsorts in the Marvin test scripts.

We'll look to add a little move configurability (such as template locations)

I also want to get working on expanding marvin to include new features and continue 'encouraging' people fixing bugs to write tests to ensure fixes 'work' and to avoid regressions later on.




Paul Angus
VP Technology   ,       ShapeBlue


t:      @cloudyangus<te...@cloudyangus>

e:      paul.angus@shapeblue.com<ma...@shapeblue.com>        |      w:      www.shapeblue.com<http://www.shapeblue.com>





-----Original Message-----
From: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Sent: 28 January 2016 17:57
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi Paul,

Indeed not all Marvin tests are generic enough and can be run on all environments. If you use this [1] collection of ~85 tests it should work well as we tested the PRs with these. Sure we should add more, but it’s probably way faster if you’d use these and work from there, instead of trying to get them all to work.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh (the name of the file by now is a bit mis-leading)



From: Paul Angus <pa...@shapeblue.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday 28 January 2016 09:29
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: RE: [PROPOSAL] Freeze everything until we get CI

Hi,

I mentioned last week I am working with a few of my colleagues to develop a repo of the Ansible plays to be able deploy both Marvin and virtualised environments - (relatively) simply and then run Marvin against that environment.

As I said, we have a load of work to do to get to the bottom of issues, as currently a large proportion of the 250 Marvin tests fail in my 'early' builds.

Once we triage issues into Marvin build/Marvin Tests/CloudStack Env/CloudStack Code, we'll be in better shape to see where we are.

Our intention is to not only share our approach and code, but also (with everyone's blessing) to work to integrate it into the infra that Citrix donated to Apache and allow anyone to ask this 'communal marvin' to generate a specified environment and run any of the integration tests against master or any given PR....

We are actively working on this, but it is going to take a little time....




[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology , ShapeBlue


d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784>

e: paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com>

a: 53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imagec996fc.png@836cfe96.47bdedb6]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
Sent: 28 January 2016 14:45
To: <de...@cloudstack.apache.org>> <de...@cloudstack.apache.org>>
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi,

I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI, If someone wants to implement their own CI, the link below gives one way to do it.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration

Thanks,
Bharat.

On 28-Jan-2016, at 2:52 AM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 10:02 PM, Remi Bergsma wrote:
Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
- the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs


We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.


Yes, that's great code. I still have to master it, but the README should get us there.

I still need to create one which also spins up a Ceph cluster. Probably a good thing to do now ;)

Wido

People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/






On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com>> wrote:


On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.


I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional slave?

I haven't figured out the Integration tests completely personally.


In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

Wido

-Sebastien

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Remi Bergsma <RB...@schubergphilis.com>.

Hi Paul,

Indeed not all Marvin tests are generic enough and can be run on all environments. If you use this [1] collection of ~85 tests it should work well as we tested the PRs with these. Sure we should add more, but it’s probably way faster if you’d use these and work from there, instead of trying to get them all to work.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh (the name of the file by now is a bit mis-leading)



From: Paul Angus <pa...@shapeblue.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday 28 January 2016 09:29
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: RE: [PROPOSAL] Freeze everything until we get CI

Hi,

I mentioned last week I am working with a few of my colleagues to develop a repo of the Ansible plays to be able deploy both Marvin and virtualised environments - (relatively) simply and then run Marvin against that environment.

As I said, we have a load of work to do to get to the bottom of issues, as currently a large proportion of the 250 Marvin tests fail in my 'early' builds.

Once we triage issues into Marvin build/Marvin Tests/CloudStack Env/CloudStack Code, we'll be in better shape to see where we are.

Our intention is to not only share our approach and code, but also (with everyone's blessing) to work to integrate it into the infra that Citrix donated to Apache and allow anyone to ask this 'communal marvin' to generate a specified environment and run any of the integration tests against master or any given PR....

We are actively working on this, but it is going to take a little time....




[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imagec996fc.png@836cfe96.47bdedb6]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
Sent: 28 January 2016 14:45
To: <de...@cloudstack.apache.org>> <de...@cloudstack.apache.org>>
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi,

I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI, If someone wants to implement their own CI, the link below gives one way to do it.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration

Thanks,
Bharat.

On 28-Jan-2016, at 2:52 AM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 10:02 PM, Remi Bergsma wrote:
Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
- the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs


We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.


Yes, that's great code. I still have to master it, but the README should get us there.

I still need to create one which also spins up a Ceph cluster. Probably a good thing to do now ;)

Wido

People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/






On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com>> wrote:


On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.


I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional slave?

I haven't figured out the Integration tests completely personally.


In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

Wido

-Sebastien

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

RE: [PROPOSAL] Freeze everything until we get CI

Posted by Paul Angus <pa...@shapeblue.com>.

Hi,

I mentioned last week I am working with a few of my colleagues to develop a repo of the Ansible plays to be able deploy both Marvin and virtualised environments - (relatively) simply and then run Marvin against that environment.

As I said, we have a load of work to do to get to the bottom of issues, as currently a large proportion of the 250 Marvin tests fail in my 'early' builds.

Once we triage issues into Marvin build/Marvin Tests/CloudStack Env/CloudStack Code, we'll be in better shape to see where we are.

Our intention is to not only share our approach and code, but also (with everyone's blessing) to work to integrate it into the infra that Citrix donated to Apache and allow anyone to ask this 'communal marvin' to generate a specified environment and run any of the integration tests against master or any given PR....

We are actively working on this, but it is going to take a little time....




[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imagec996fc.png@836cfe96.47bdedb6]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
Sent: 28 January 2016 14:45
To: <de...@cloudstack.apache.org> <de...@cloudstack.apache.org>
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi,

I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI, If someone wants to implement their own CI, the link below gives one way to do it.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration

Thanks,
Bharat.

On 28-Jan-2016, at 2:52 AM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 10:02 PM, Remi Bergsma wrote:
Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
- the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs


We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.


Yes, that's great code. I still have to master it, but the README should get us there.

I still need to create one which also spins up a Ceph cluster. Probably a good thing to do now ;)

Wido

People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/






On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com>> wrote:


On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl>> wrote:



On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.


I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional slave?

I haven't figured out the Integration tests completely personally.


In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

Wido

-Sebastien

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Bharat Kumar <bh...@citrix.com>.

Hi,

I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI, If someone wants to implement their own CI, the link below gives one way to do it.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration

Thanks,
Bharat.

On 28-Jan-2016, at 2:52 AM, Wido den Hollander <wi...@widodh.nl>> wrote:

On 01/27/2016 10:02 PM, Remi Bergsma wrote:
Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
- the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs

We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.

Yes, that's great code. I still have to master it, but the README should
get us there.

I still need to create one which also spins up a Ceph cluster. Probably
a good thing to do now ;)

Wido

People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/

On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com>> wrote:

On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl>> wrote:

On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.

I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional
slave?

I haven't figured out the Integration tests completely personally.

In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

Wido

-Sebastien

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Wido den Hollander <wi...@widodh.nl>.


On 01/27/2016 10:02 PM, Remi Bergsma wrote:
> Hi all,
> 
> We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).
> 
> If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.
> 
> In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
> - there is no community hardware to run integration tests on (there was supposed to be something)
> - no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
> - the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs
> 
> 
> We tried to address points 1 and 2 and I propose to give up on them.
> 
> If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.
> 

Yes, that's great code. I still have to master it, but the README should
get us there.

I still need to create one which also spins up a Ceph cluster. Probably
a good thing to do now ;)

Wido

> People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.
> 
> This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.
> 
> Regards,
> Remi
> 
> [1] https://github.com/schubergphilis/MCT-shared/
> 
> 
> 
> 
> 
> 
> On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com> wrote:
> 
>>
>>> On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>>
>>>
>>> On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
>>>> Folks,
>>>>
>>>> How about we freeze our repo entirely until we get proper CI in place.
>>>>
>>>> Seems to me all the hard work from Remi and co could be lost if we start committing again.
>>>>
>>>> Now Travis is not running, Jenkins fails all the time and nobody cares…
>>>>
>>>> So how about we figure out CI now ? and not do anything else.
>>>>
>>>
>>> I think forces have to be combined to make this work.
>>>
>>> Questions which come to mind: Who runs Jenkins? Do we need a additional
>>> slave?
>>>
>>> I haven't figured out the Integration tests completely personally.
>>>
>>
>> In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
>> Only when all are green can we merge.
>>
>> there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.
>>
>> for instance, how can pcextreme automate its testing and report back on each PR ?
>>
>>> Wido
>>>
>>>> -Sebastien
>>>>
>>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Remi Bergsma <RB...@schubergphilis.com>.

Sure Rohit, sometimes it makes no sense to run integration tests. That’s why for example I build an environment and check the UI when someone changes a javascript file.

Just saying it “looks” good is valuable from a code review point of view, but not from a verification point of view. I say we always need both.

As a general rule, every PR should be “tested” so one way or the other you need to show it works. Can be integration tests, can be UI screenshot, can be successful package build result or whatever. Currently, there is not much real “testing” going on and people expect this to magically happen somehow by someone else. Currently I feel quite alone in performing that "magic". I can’t and won’t keep that up.

Regards,
Remi


From: Rohit Yadav <ro...@shapeblue.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday 28 January 2016 09:50
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: Re: [PROPOSAL] Freeze everything until we get CI



[ShapeBlue]<http://www.shapeblue.com>
Rohit Yadav
Software Architect      ,       ShapeBlue


d:       | s: +44 203 603 0540<tel:|%20s:%20+44%20203%20603%200540>      |      m:      +91 8826230892<tel:+91%208826230892>

e:      rohit.yadav@shapeblue.com | t: <mailto:rohit.yadav@shapeblue.com%20|%20t:>       |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:image24a672.png@c43ecb78.4d9cef18]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




On 27-Jan-2016, at 10:02 PM, Remi Bergsma <RB...@schubergphilis.com>> wrote:

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

In general, a good idea though integration tests won’t apply everywhere to help us maintain quality.

We can cheat by avoiding integration tests but do other kinds of tests (manually for now) based on that Travis/Jenkins strictly pass and diffs are entirely:

- Text based in API doc/params for example, README files, comments, etc.

- Non-java code, not even systemvm related scripts where they may require alternative tests; such as changes in systemvm-template building scripts, marvin, docker etc.

- Javascript file changes, as that would require some manual testing and screenshots and won’t be caught by integration tests

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rohit Yadav <ro...@shapeblue.com>.


[ShapeBlue]<http://www.shapeblue.com>
Rohit Yadav
Software Architect      ,       ShapeBlue


d:       | s: +44 203 603 0540<tel:|%20s:%20+44%20203%20603%200540>      |      m:      +91 8826230892<tel:+91%208826230892>

e:      rohit.yadav@shapeblue.com | t: <mailto:rohit.yadav@shapeblue.com%20|%20t:>       |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:image24a672.png@c43ecb78.4d9cef18]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




On 27-Jan-2016, at 10:02 PM, Remi Bergsma <RB...@schubergphilis.com>> wrote:

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

In general, a good idea though integration tests won’t apply everywhere to help us maintain quality.

We can cheat by avoiding integration tests but do other kinds of tests (manually for now) based on that Travis/Jenkins strictly pass and diffs are entirely:

- Text based in API doc/params for example, README files, comments, etc.

- Non-java code, not even systemvm related scripts where they may require alternative tests; such as changes in systemvm-template building scripts, marvin, docker etc.

- Javascript file changes, as that would require some manual testing and screenshots and won’t be caught by integration tests

Regards.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Remi Bergsma <RB...@schubergphilis.com>.

Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things may look OK, but still break something else in an unexpected way. IMHO, giving up on it would be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s nowhere close.
- the community has very few people that are able to run the integration tests on their own laptop/hardware and post the results on PRs

We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that can run KVM, then clone the code we use [1] it should be pretty straight forward to run the integration tests. We choose to make it all virtual and used nested-virtualisation, but that is optional. A simple Intel Nuc or similar will do.

People can then test the PRs that they find interesting and post results, after which it can be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/

On 27/01/16 21:33, "Sebastien Goasguen" <ru...@gmail.com> wrote:

>
>> On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>> 
>> 
>> 
>> On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
>>> Folks,
>>> 
>>> How about we freeze our repo entirely until we get proper CI in place.
>>> 
>>> Seems to me all the hard work from Remi and co could be lost if we start committing again.
>>> 
>>> Now Travis is not running, Jenkins fails all the time and nobody cares…
>>> 
>>> So how about we figure out CI now ? and not do anything else.
>>> 
>> 
>> I think forces have to be combined to make this work.
>> 
>> Questions which come to mind: Who runs Jenkins? Do we need a additional
>> slave?
>> 
>> I haven't figured out the Integration tests completely personally.
>> 
>
>In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
>Only when all are green can we merge.
>
>there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.
>
>for instance, how can pcextreme automate its testing and report back on each PR ?
>
>> Wido
>> 
>>> -Sebastien
>>> 
>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Remi Bergsma <RB...@schubergphilis.com>.

This won’t fly, you will get no hook approved.

Instead, simply poll the Github API every x minutes for new PRs and then start your test.

Regards,
Remi




On 27/01/16 21:59, "Rafael Weingärtner" <ra...@gmail.com> wrote:

>Because that would require manual work (reporting back the results), maybe
>if we figure out some way to dispatch the request to run tests with a PR
>against an environment.
>
>If we had a repository in which developers could register their environment
>to receive requests to run tests on a given PR, and then store the result
>somewhere to be checked by some git hook.
>
>When a PR is created a hook dispatches test requests, and then other hook
>checks the results of those tests (instead of reporting back, we consult
>the result of functional software tests).

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rafael Weingärtner <ra...@gmail.com>.

Because that would require manual work (reporting back the results), maybe
if we figure out some way to dispatch the request to run tests with a PR
against an environment.

If we had a repository in which developers could register their environment
to receive requests to run tests on a given PR, and then store the result
somewhere to be checked by some git hook.

When a PR is created a hook dispatches test requests, and then other hook
checks the results of those tests (instead of reporting back, we consult
the result of functional software tests).

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Sebastien Goasguen <ru...@gmail.com>.

> On Jan 27, 2016, at 9:42 PM, Rafael Weingärtner <ra...@gmail.com> wrote:
> 
> +1 Sebastian,
> I would ask the same as Wilder, why the rush to merge those PRs without
> actual test and reviews (some LGTMs there I would not count)? They did not
> seem to be that important.
> 
> I believe that the idea to run PRs totally distributed in everyone’s
> environments would not be feasible today. However, we could have a single
> environment with the most common environment configurations such as a
> XenServer cluster, KVM cluster, LXC cluster and so forth, so we can run
> functional tests (the so-called integration tests that Remi is running
> today) against that environment. We could have both advanced and basic
> network configurations environments; I think that a single place to
> concentrate our efforts would be better today.
> 

we have failed miserably at making this happen centrally.

we might have more luck with asking everyone to setup a cloudstack box of their preferred setup and asking them to run marvin tests, then report centrally.
I don’t see why it would not be feasible.

> On Wed, Jan 27, 2016 at 6:33 PM, Sebastien Goasguen <ru...@gmail.com>
> wrote:
> 
>> 
>>> On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>>> 
>>> 
>>> 
>>> On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
>>>> Folks,
>>>> 
>>>> How about we freeze our repo entirely until we get proper CI in place.
>>>> 
>>>> Seems to me all the hard work from Remi and co could be lost if we
>> start committing again.
>>>> 
>>>> Now Travis is not running, Jenkins fails all the time and nobody cares…
>>>> 
>>>> So how about we figure out CI now ? and not do anything else.
>>>> 
>>> 
>>> I think forces have to be combined to make this work.
>>> 
>>> Questions which come to mind: Who runs Jenkins? Do we need a additional
>>> slave?
>>> 
>>> I haven't figured out the Integration tests completely personally.
>>> 
>> 
>> In an ideal case, PR should trigger tests totally distributed on
>> everyone’s own hardware. Then tests would report back on the PR.
>> Only when all are green can we merge.
>> 
>> there is an issue with creating triggers in github on our own, but I think
>> that’s what we should aspire to.
>> 
>> for instance, how can pcextreme automate its testing and report back on
>> each PR ?
>> 
>>> Wido
>>> 
>>>> -Sebastien
>>>> 
>> 
>> 
> 
> 
> -- 
> Rafael Weingärtner

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Rafael Weingärtner <ra...@gmail.com>.

+1 Sebastian,
I would ask the same as Wilder, why the rush to merge those PRs without
actual test and reviews (some LGTMs there I would not count)? They did not
seem to be that important.

I believe that the idea to run PRs totally distributed in everyone’s
environments would not be feasible today. However, we could have a single
environment with the most common environment configurations such as a
XenServer cluster, KVM cluster, LXC cluster and so forth, so we can run
functional tests (the so-called integration tests that Remi is running
today) against that environment. We could have both advanced and basic
network configurations environments; I think that a single place to
concentrate our efforts would be better today.

On Wed, Jan 27, 2016 at 6:33 PM, Sebastien Goasguen <ru...@gmail.com>
wrote:

>
> > On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl> wrote:
> >
> >
> >
> > On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
> >> Folks,
> >>
> >> How about we freeze our repo entirely until we get proper CI in place.
> >>
> >> Seems to me all the hard work from Remi and co could be lost if we
> start committing again.
> >>
> >> Now Travis is not running, Jenkins fails all the time and nobody cares…
> >>
> >> So how about we figure out CI now ? and not do anything else.
> >>
> >
> > I think forces have to be combined to make this work.
> >
> > Questions which come to mind: Who runs Jenkins? Do we need a additional
> > slave?
> >
> > I haven't figured out the Integration tests completely personally.
> >
>
> In an ideal case, PR should trigger tests totally distributed on
> everyone’s own hardware. Then tests would report back on the PR.
> Only when all are green can we merge.
>
> there is an issue with creating triggers in github on our own, but I think
> that’s what we should aspire to.
>
> for instance, how can pcextreme automate its testing and report back on
> each PR ?
>
> > Wido
> >
> >> -Sebastien
> >>
>
>

-- 
Rafael Weingärtner

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Sebastien Goasguen <ru...@gmail.com>.

> On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wi...@widodh.nl> wrote:
> 
> 
> 
> On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
>> Folks,
>> 
>> How about we freeze our repo entirely until we get proper CI in place.
>> 
>> Seems to me all the hard work from Remi and co could be lost if we start committing again.
>> 
>> Now Travis is not running, Jenkins fails all the time and nobody cares…
>> 
>> So how about we figure out CI now ? and not do anything else.
>> 
> 
> I think forces have to be combined to make this work.
> 
> Questions which come to mind: Who runs Jenkins? Do we need a additional
> slave?
> 
> I haven't figured out the Integration tests completely personally.
> 

In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware. Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

> Wido
> 
>> -Sebastien
>>

Re: [PROPOSAL] Freeze everything until we get CI

Posted by Wido den Hollander <wi...@widodh.nl>.

On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
> Folks,
> 
> How about we freeze our repo entirely until we get proper CI in place.
> 
> Seems to me all the hard work from Remi and co could be lost if we start committing again.
> 
> Now Travis is not running, Jenkins fails all the time and nobody cares…
> 
> So how about we figure out CI now ? and not do anything else.
> 

I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional
slave?

I haven't figured out the Integration tests completely personally.

Wido

> -Sebastien
>