You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Alex Huang <Al...@citrix.com> on 2013/06/29 02:18:17 UTC

[DISCUSS] If BVT breaks, revert the commits...

After Dave's complain in the vmsync [MERGE] thread about BVT in horrible shape on master, I went around to figure out what exactly happened.  The best I can figure is that after a certain merge (I will leave out which merge as that's not important), BVT no longer runs automatically.  It was promised to be fixed and there are people who are actively fixing it but it's been in this way for about two weeks.  People running BVTs are working around the problem but it's not automated anymore and so it's no longer running on master.  I understand people are nice and tried to be accommodating to other people by working around the problem but sometimes we just have to be an arse.  So let me be that arse...

New Rule....
If BVT or automated regression tests break on master or any release branch, we revert all commits that broke it.  It doesn't matter if they promise to fix it within the next hour.  If it's broken, the release manager will revert the commits and developers must resubmit.  It sounds mean but it's the only way this problem can be fixed.  

To avoid having a bunch of reverts and resubmits, the developers should be able to request that BVT run on their branch and don't merge until BVT on their branch is at 100%.  We will work on figuring out how to do that.

Comments?

--Alex

Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Jul 15, 2013 at 03:06:41PM +0530, Girish Shilamkar wrote:
> Alex,
> 
> On 10-Jul-2013, at 9:32 AM, Alex Huang <Al...@citrix.com>
> wrote:
> 
> > Sorry this took a little longer than expected with the holiday in
> > US.  Here's the first proposal [1] on the automated test system.
> > Comments welcome.  
> > 
> > Specifically, I have one question on if we should have a staging
> > branch for all release branches and master where all checkins go
> > and the build system automatically cherry-pick over commits that
> > passes the tests.
> +1 Staging branch is better option than to revert the commits. One
> useful way might be to trigger a jenkins job once a patch is
> uploaded to review board and run BVT.

A staging area would indeed be useful for passing tests and gating the
checkins.

> But as of now Jenkins plugin for review board does not have git
> support.
> https://wiki.jenkins-ci.org/display/JENKINS/Reviewboard+Plugin

Yeah - this is a perforce only plugin I'm afraid.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by Girish Shilamkar <gi...@clogeny.com>.
Alex,

On 10-Jul-2013, at 9:32 AM, Alex Huang <Al...@citrix.com> wrote:

> Sorry this took a little longer than expected with the holiday in US.  Here's the first proposal [1] on the automated test system.  Comments welcome.  
> 
> Specifically, I have one question on if we should have a staging branch for all release branches and master where all checkins go and the build system automatically cherry-pick over commits that passes the tests.
+1 
Staging branch is better option than to revert the commits. One useful way might be to trigger a jenkins job once a patch is uploaded to review board and run BVT.
But as of now Jenkins plugin for review board does not have git support. https://wiki.jenkins-ci.org/display/JENKINS/Reviewboard+Plugin

Regards,
Girish

> 
> I believe we have a lot of pieces in place.  Prassana, Sudha, Rayees, Ram Ganesh, myself, and a few others will be working to get this system in place.  
> 
> --Alex
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines
> 
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Friday, June 28, 2013 8:44 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [DISCUSS] If BVT breaks, revert the commits...
>> 
>> On Fri, Jun 28, 2013 at 8:18 PM, Alex Huang <Al...@citrix.com> wrote:
>>> After Dave's complain in the vmsync [MERGE] thread about BVT in horrible
>> shape on master, I went around to figure out what exactly happened.  The
>> best I can figure is that after a certain merge (I will leave out which merge as
>> that's not important), BVT no longer runs automatically.  It was promised to
>> be fixed and there are people who are actively fixing it but it's been in this
>> way for about two weeks.  People running BVTs are working around the
>> problem but it's not automated anymore and so it's no longer running on
>> master.  I understand people are nice and tried to be accommodating to
>> other people by working around the problem but sometimes we just have to
>> be an arse.  So let me be that arse...
>>> 
>>> New Rule....
>>> If BVT or automated regression tests break on master or any release
>> branch, we revert all commits that broke it.  It doesn't matter if they promise
>> to fix it within the next hour.  If it's broken, the release manager will revert
>> the commits and developers must resubmit.  It sounds mean but it's the only
>> way this problem can be fixed.
>>> 
>>> To avoid having a bunch of reverts and resubmits, the developers should
>> be able to request that BVT run on their branch and don't merge until BVT on
>> their branch is at 100%.  We will work on figuring out how to do that.
>>> 
>>> Comments?
>>> 
>>> --Alex
>> 
>> +100 - not only +100 but I will increment ASFBots $beverage counter a
>> few in your favor for suggesting this.
>> 
>> --David


RE: [DISCUSS] If BVT breaks, revert the commits...

Posted by Alex Huang <Al...@citrix.com>.
Sorry this took a little longer than expected with the holiday in US.  Here's the first proposal [1] on the automated test system.  Comments welcome.  

Specifically, I have one question on if we should have a staging branch for all release branches and master where all checkins go and the build system automatically cherry-pick over commits that passes the tests.

I believe we have a lot of pieces in place.  Prassana, Sudha, Rayees, Ram Ganesh, myself, and a few others will be working to get this system in place.  

--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines

> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Friday, June 28, 2013 8:44 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] If BVT breaks, revert the commits...
> 
> On Fri, Jun 28, 2013 at 8:18 PM, Alex Huang <Al...@citrix.com> wrote:
> > After Dave's complain in the vmsync [MERGE] thread about BVT in horrible
> shape on master, I went around to figure out what exactly happened.  The
> best I can figure is that after a certain merge (I will leave out which merge as
> that's not important), BVT no longer runs automatically.  It was promised to
> be fixed and there are people who are actively fixing it but it's been in this
> way for about two weeks.  People running BVTs are working around the
> problem but it's not automated anymore and so it's no longer running on
> master.  I understand people are nice and tried to be accommodating to
> other people by working around the problem but sometimes we just have to
> be an arse.  So let me be that arse...
> >
> > New Rule....
> > If BVT or automated regression tests break on master or any release
> branch, we revert all commits that broke it.  It doesn't matter if they promise
> to fix it within the next hour.  If it's broken, the release manager will revert
> the commits and developers must resubmit.  It sounds mean but it's the only
> way this problem can be fixed.
> >
> > To avoid having a bunch of reverts and resubmits, the developers should
> be able to request that BVT run on their branch and don't merge until BVT on
> their branch is at 100%.  We will work on figuring out how to do that.
> >
> > Comments?
> >
> > --Alex
> 
> +100 - not only +100 but I will increment ASFBots $beverage counter a
> few in your favor for suggesting this.
> 
> --David

Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by David Nalley <da...@gnsa.us>.
On Fri, Jun 28, 2013 at 8:18 PM, Alex Huang <Al...@citrix.com> wrote:
> After Dave's complain in the vmsync [MERGE] thread about BVT in horrible shape on master, I went around to figure out what exactly happened.  The best I can figure is that after a certain merge (I will leave out which merge as that's not important), BVT no longer runs automatically.  It was promised to be fixed and there are people who are actively fixing it but it's been in this way for about two weeks.  People running BVTs are working around the problem but it's not automated anymore and so it's no longer running on master.  I understand people are nice and tried to be accommodating to other people by working around the problem but sometimes we just have to be an arse.  So let me be that arse...
>
> New Rule....
> If BVT or automated regression tests break on master or any release branch, we revert all commits that broke it.  It doesn't matter if they promise to fix it within the next hour.  If it's broken, the release manager will revert the commits and developers must resubmit.  It sounds mean but it's the only way this problem can be fixed.
>
> To avoid having a bunch of reverts and resubmits, the developers should be able to request that BVT run on their branch and don't merge until BVT on their branch is at 100%.  We will work on figuring out how to do that.
>
> Comments?
>
> --Alex

+100 - not only +100 but I will increment ASFBots $beverage counter a
few in your favor for suggesting this.

--David

RE: [DISCUSS] If BVT breaks, revert the commits...

Posted by Alex Huang <Al...@citrix.com>.
> This only thing i'm a bit afraid of is the technical side of reverting a merge. It's
> one of those this that you need to be careful with in git.

I think this means all merges must come in as one big squashed patch.  If they come in as multiple commits or rebase, then it gets reverted automatically.

--Alex

Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by Hugo Trippaers <hu...@trippaers.nl>.
Alex,

+1

From the jenkins side we could make more jobs with parameters so anyone can kick-off a test run on his/her branch. I don't know if that would work for the BVT tests though.

This only thing i'm a bit afraid of is the technical side of reverting a merge. It's one of those this that you need to be careful with in git.

Cheers,

Hugo



On Jun 28, 2013, at 5:18 PM, Alex Huang <Al...@citrix.com> wrote:

> After Dave's complain in the vmsync [MERGE] thread about BVT in horrible shape on master, I went around to figure out what exactly happened.  The best I can figure is that after a certain merge (I will leave out which merge as that's not important), BVT no longer runs automatically.  It was promised to be fixed and there are people who are actively fixing it but it's been in this way for about two weeks.  People running BVTs are working around the problem but it's not automated anymore and so it's no longer running on master.  I understand people are nice and tried to be accommodating to other people by working around the problem but sometimes we just have to be an arse.  So let me be that arse...
> 
> New Rule....
> If BVT or automated regression tests break on master or any release branch, we revert all commits that broke it.  It doesn't matter if they promise to fix it within the next hour.  If it's broken, the release manager will revert the commits and developers must resubmit.  It sounds mean but it's the only way this problem can be fixed.  
> 
> To avoid having a bunch of reverts and resubmits, the developers should be able to request that BVT run on their branch and don't merge until BVT on their branch is at 100%.  We will work on figuring out how to do that.
> 
> Comments?
> 
> --Alex


Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jun 29, 2013 at 12:31:31PM -0400, David Nalley wrote:
> On Sat, Jun 29, 2013 at 12:45 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> > On Sat, Jun 29, 2013 at 12:18:17AM +0000, Alex Huang wrote:
> >>
> >> To avoid having a bunch of reverts and resubmits, the developers
> >> should be able to request that BVT run on their branch and don't
> >> merge until BVT on their branch is at 100%.  We will work on
> >> figuring out how to do that.
> >>
> >> Comments?
> >>
> > Quick questions:
> >
> > 1. How do we test contributor code (from github/reviewboard) because
> > we have no staging area/branch?
> >
> 
> Take a look at the pre-commit testing that hadoop does, that might be
> usable for us. Olamy did a blog post about it recently.
> 

Thanks for the pointers! I will check it out.

> >
> > 4. Who will fix the tests which are broken?
> >
> 
> Well if we treat master as frozen until the tests are fixed, then I'd
> suppose that folks wanting to get code in will be incentivized to at
> least help fix them so that master can start getting code again :)
> 

It's an imposition not an incentive then :)

The reason for the impossibly long fix cycles for automated tests is
mainly because there isn't any review of test plans and tests
automated from those plans. So the project coming to Apache to *me*
personally has been a boon in this way to fix this process failure. At
least now we have reviews and a few more eyes looking at the problems
... even more would help.

Once we get to fixing the existing tests I think everyone will begin
to see how bloated some of them can get and can be simplified from an
automated perspective.

So I'm in on Alex's plan, in fact I fix tests everyday so glad to have
others join the effort.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by David Nalley <da...@gnsa.us>.
On Sat, Jun 29, 2013 at 12:45 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> On Sat, Jun 29, 2013 at 12:18:17AM +0000, Alex Huang wrote:
>> After Dave's complain in the vmsync [MERGE] thread about BVT in
>> horrible shape on master, I went around to figure out what exactly
>> happened.  The best I can figure is that after a certain merge (I
>> will leave out which merge as that's not important), BVT no longer
>> runs automatically.  It was promised to be fixed and there are
>> people who are actively fixing it but it's been in this way for
>> about two weeks.  People running BVTs are working around the problem
>> but it's not automated anymore and so it's no longer running on
>> master.  I understand people are nice and tried to be accommodating
>> to other people by working around the problem but sometimes we just
>> have to be an arse.  So let me be that arse...
>>
>> New Rule....
>> If BVT or automated regression tests break on master or any release
>> branch, we revert all commits that broke it.  It doesn't matter if
>> they promise to fix it within the next hour.  If it's broken, the
>> release manager will revert the commits and developers must
>> resubmit.  It sounds mean but it's the only way this problem can be
>> fixed.
>>
>> To avoid having a bunch of reverts and resubmits, the developers
>> should be able to request that BVT run on their branch and don't
>> merge until BVT on their branch is at 100%.  We will work on
>> figuring out how to do that.
>>
>> Comments?
>>
> Quick questions:
>
> 1. How do we test contributor code (from github/reviewboard) because
> we have no staging area/branch?
>

Take a look at the pre-commit testing that hadoop does, that might be
usable for us. Olamy did a blog post about it recently.


> 2. Squashing into a big patch is not a good idea. I prefer that we
> keep the clean commit history rebased atop master in the topic branch
> and run the bvt on it. This can be requested and there is ability in
> the test jobs to switch branches already.
>
> Hit Build (test-matrix job) -> choose <topic-branch> on the form.
>
> 3. BVTs have long runtime (3 HVs - 4hrs). For large merges it makes
> sense since we're going to be waiting 72h for the review/merge
> request. But rest of the time I propose we run it against master only.
> Is this the same as you're proposing?
>
> 4. Who will fix the tests which are broken?
>

Well if we treat master as frozen until the tests are fixed, then I'd
suppose that folks wanting to get code in will be incentivized to at
least help fix them so that master can start getting code again :)

--David

Re: [DISCUSS] If BVT breaks, revert the commits...

Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jun 29, 2013 at 12:18:17AM +0000, Alex Huang wrote:
> After Dave's complain in the vmsync [MERGE] thread about BVT in
> horrible shape on master, I went around to figure out what exactly
> happened.  The best I can figure is that after a certain merge (I
> will leave out which merge as that's not important), BVT no longer
> runs automatically.  It was promised to be fixed and there are
> people who are actively fixing it but it's been in this way for
> about two weeks.  People running BVTs are working around the problem
> but it's not automated anymore and so it's no longer running on
> master.  I understand people are nice and tried to be accommodating
> to other people by working around the problem but sometimes we just
> have to be an arse.  So let me be that arse...
> 
> New Rule....
> If BVT or automated regression tests break on master or any release
> branch, we revert all commits that broke it.  It doesn't matter if
> they promise to fix it within the next hour.  If it's broken, the
> release manager will revert the commits and developers must
> resubmit.  It sounds mean but it's the only way this problem can be
> fixed.  
> 
> To avoid having a bunch of reverts and resubmits, the developers
> should be able to request that BVT run on their branch and don't
> merge until BVT on their branch is at 100%.  We will work on
> figuring out how to do that.
> 
> Comments?
> 
Quick questions:

1. How do we test contributor code (from github/reviewboard) because
we have no staging area/branch?

2. Squashing into a big patch is not a good idea. I prefer that we
keep the clean commit history rebased atop master in the topic branch
and run the bvt on it. This can be requested and there is ability in
the test jobs to switch branches already. 

Hit Build (test-matrix job) -> choose <topic-branch> on the form.

3. BVTs have long runtime (3 HVs - 4hrs). For large merges it makes
sense since we're going to be waiting 72h for the review/merge
request. But rest of the time I propose we run it against master only.
Is this the same as you're proposing?

4. Who will fix the tests which are broken?

-- 
Prasanna.,

------------------------
Powered by BigRock.com