You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by David Nalley <da...@gnsa.us> on 2013/02/18 02:43:47 UTC
How we treat broken builds.
Hi folks,
I know folks have said this elsewhere, but I'd like to reiterate it.
I am somewhat frustrated with our reaction to broken builds. In
general it seems we don't care, and this makes it more difficult to
fix problems. Jenkins reporting a broken build (be it a broken run of
RAT, failure to compile, failure of a unit test, building docs, etc.)
should be our Andon cord [1]. We should all stop commits that aren't
fixing the broken build. To illustrate why this is a problem,
RAT failures started occurring recently, this keeps us from testing
whether CloudStack builds, because each build is conditioned on the
successful completion of the test before it. That in turn keeps
apidocs from building, which keeps marvin from building, which keeps
documentation from building. We essentially are blind until it gets
fixed.
That means, that like TPS, when the andon cord gets pulled we all need
to focus on the problem, and not continuing our own work and ignoring
the problem and potentially contributing to making the solution more
painful.
My requests:
If you see that the branch you are working on is broken - please don't
commit to it, unless your commit is going to fix it.
If you see that the branch you are working on is broken - please help
fix it. (This should become priority #1, for everyone)
Please don't make a commit the last thing you do before going offline
- make sure that your commit isn't breaking any of the tests before
you leave. If your commit breaks something and you've gone offline for
16 hours, you've made life painful for others, so make sure there is
some buffer between your last commit and you going offline so you can
see and remedy any problems that arise.
And just for the record - as of this moment, 4.1 and master are broken
at various stages.
--David
[1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
Re: How we treat broken builds.
Posted by Dave Cahill <dc...@midokura.com>.
Strongly agree. Master was very painful to work with last week.
On Mon, Feb 18, 2013 at 10:43 AM, David Nalley <da...@gnsa.us> wrote:
> Hi folks,
>
> I know folks have said this elsewhere, but I'd like to reiterate it.
>
> I am somewhat frustrated with our reaction to broken builds. In
> general it seems we don't care, and this makes it more difficult to
> fix problems. Jenkins reporting a broken build (be it a broken run of
> RAT, failure to compile, failure of a unit test, building docs, etc.)
> should be our Andon cord [1]. We should all stop commits that aren't
> fixing the broken build. To illustrate why this is a problem,
>
> RAT failures started occurring recently, this keeps us from testing
> whether CloudStack builds, because each build is conditioned on the
> successful completion of the test before it. That in turn keeps
> apidocs from building, which keeps marvin from building, which keeps
> documentation from building. We essentially are blind until it gets
> fixed.
>
> That means, that like TPS, when the andon cord gets pulled we all need
> to focus on the problem, and not continuing our own work and ignoring
> the problem and potentially contributing to making the solution more
> painful.
>
> My requests:
>
> If you see that the branch you are working on is broken - please don't
> commit to it, unless your commit is going to fix it.
> If you see that the branch you are working on is broken - please help
> fix it. (This should become priority #1, for everyone)
> Please don't make a commit the last thing you do before going offline
> - make sure that your commit isn't breaking any of the tests before
> you leave. If your commit breaks something and you've gone offline for
> 16 hours, you've made life painful for others, so make sure there is
> some buffer between your last commit and you going offline so you can
> see and remedy any problems that arise.
>
> And just for the record - as of this moment, 4.1 and master are broken
> at various stages.
>
>
> --David
>
> [1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
>
Re: How we treat broken builds.
Posted by David Nalley <da...@gnsa.us>.
On Wed, Feb 20, 2013 at 1:07 AM, Animesh Chaturvedi
<an...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Sunday, February 17, 2013 7:32 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: How we treat broken builds.
>>
>> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com>
>> wrote:
>> > Are there jenkins reports the committers can subscribe to and get
>> > email when there are breakages? I know at least once 4.1 was broken
>> > while I was in the process of testing a commit (just a quick sanity
>> > check I do to avoid being the guy who causes these sorts of things),
>> > and then I committed on top of that broken branch. If there are such
>> > emails (and I suspect there are), it would help me to know when to
>> > stop working on the branch, and all of the committers should probably sign
>> up.
>> > On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
>> >
>>
>>
>> Yes there are - the commits mailing list receives the notifications from both
>> jenkins.cs.o and builds.a.o.
>> Thanks for asking the question, I just assumed everyone knew of, and was
>> subscribed to the mailing list.
>>
>> You can subscribe by sending an email to:
>>
>> cloudstack-commits-subscribe@incubator.apache.org
>>
>> --David
> [Animesh>] David is there a way to check how many folks are subscribed to commit email list? If it is way out of whack with the number of folks subscribed to main mailing list then we need to educate folks
>
Yes, any of the moderators can get a list of all of the folks subscribed.
I'll do this today and run through the list to see what committers
aren't subscribed.
--David
RE: How we treat broken builds.
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Sunday, February 17, 2013 7:32 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: How we treat broken builds.
>
> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com>
> wrote:
> > Are there jenkins reports the committers can subscribe to and get
> > email when there are breakages? I know at least once 4.1 was broken
> > while I was in the process of testing a commit (just a quick sanity
> > check I do to avoid being the guy who causes these sorts of things),
> > and then I committed on top of that broken branch. If there are such
> > emails (and I suspect there are), it would help me to know when to
> > stop working on the branch, and all of the committers should probably sign
> up.
> > On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
> >
>
>
> Yes there are - the commits mailing list receives the notifications from both
> jenkins.cs.o and builds.a.o.
> Thanks for asking the question, I just assumed everyone knew of, and was
> subscribed to the mailing list.
>
> You can subscribe by sending an email to:
>
> cloudstack-commits-subscribe@incubator.apache.org
>
> --David
[Animesh>] David is there a way to check how many folks are subscribed to commit email list? If it is way out of whack with the number of folks subscribed to main mailing list then we need to educate folks
Re: How we treat broken builds.
Posted by Chip Childers <ch...@sungard.com>.
On Wed, Feb 20, 2013 at 09:53:43AM -0500, David Nalley wrote:
> On Tue, Feb 19, 2013 at 11:32 AM, Chip Childers
> <ch...@sungard.com> wrote:
> > On Sun, Feb 17, 2013 at 10:31:44PM -0500, David Nalley wrote:
> >> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> >> > Are there jenkins reports the committers can subscribe to and get email
> >> > when there are breakages? I know at least once 4.1 was broken while I was
> >> > in the process of testing a commit (just a quick sanity check I do to avoid
> >> > being the guy who causes these sorts of things), and then I committed on
> >> > top of that broken branch. If there are such emails (and I suspect there
> >> > are), it would help me to know when to stop working on the branch, and all
> >> > of the committers should probably sign up.
> >> > On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
> >> >
> >>
> >>
> >> Yes there are - the commits mailing list receives the notifications
> >> from both jenkins.cs.o and builds.a.o.
> >> Thanks for asking the question, I just assumed everyone knew of, and
> >> was subscribed to the mailing list.
> >>
> >> You can subscribe by sending an email to:
> >>
> >> cloudstack-commits-subscribe@incubator.apache.org
> >>
> >> --David
> >>
> >
> > I've noticed that although builds.a.o is configured to email the
> > committers when the build breaks for their changes, it's sending to
> > weird addresses. For example, I see it sending notices to
> > chip.childers@apache.org, which doesn't exist. It seems to take the
> > username from the committer email address, and then use @a.o. Weird.
> >
> > Also, perhaps we can configure the jobs to email build failures and "all
> > better" emails to the dev list instead of commits? I know that
> > committers should be watching the commits, but I bet that doesn't happen
> > in real time... and that the dev list would be a better place to get
> > attention a little more quickly.
> >
>
>
> That is weird. Probably an artifact of how userids are treated in SVN.
> Something to bring up with builds@.
> I am okay with that change - I assume you are going to make it?
>
> --David
>
Yes, I'll email builds@ now.
Re: How we treat broken builds.
Posted by David Nalley <da...@gnsa.us>.
On Tue, Feb 19, 2013 at 11:32 AM, Chip Childers
<ch...@sungard.com> wrote:
> On Sun, Feb 17, 2013 at 10:31:44PM -0500, David Nalley wrote:
>> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com> wrote:
>> > Are there jenkins reports the committers can subscribe to and get email
>> > when there are breakages? I know at least once 4.1 was broken while I was
>> > in the process of testing a commit (just a quick sanity check I do to avoid
>> > being the guy who causes these sorts of things), and then I committed on
>> > top of that broken branch. If there are such emails (and I suspect there
>> > are), it would help me to know when to stop working on the branch, and all
>> > of the committers should probably sign up.
>> > On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
>> >
>>
>>
>> Yes there are - the commits mailing list receives the notifications
>> from both jenkins.cs.o and builds.a.o.
>> Thanks for asking the question, I just assumed everyone knew of, and
>> was subscribed to the mailing list.
>>
>> You can subscribe by sending an email to:
>>
>> cloudstack-commits-subscribe@incubator.apache.org
>>
>> --David
>>
>
> I've noticed that although builds.a.o is configured to email the
> committers when the build breaks for their changes, it's sending to
> weird addresses. For example, I see it sending notices to
> chip.childers@apache.org, which doesn't exist. It seems to take the
> username from the committer email address, and then use @a.o. Weird.
>
> Also, perhaps we can configure the jobs to email build failures and "all
> better" emails to the dev list instead of commits? I know that
> committers should be watching the commits, but I bet that doesn't happen
> in real time... and that the dev list would be a better place to get
> attention a little more quickly.
>
That is weird. Probably an artifact of how userids are treated in SVN.
Something to bring up with builds@.
I am okay with that change - I assume you are going to make it?
--David
Re: How we treat broken builds.
Posted by Chip Childers <ch...@sungard.com>.
On Sun, Feb 17, 2013 at 10:31:44PM -0500, David Nalley wrote:
> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> > Are there jenkins reports the committers can subscribe to and get email
> > when there are breakages? I know at least once 4.1 was broken while I was
> > in the process of testing a commit (just a quick sanity check I do to avoid
> > being the guy who causes these sorts of things), and then I committed on
> > top of that broken branch. If there are such emails (and I suspect there
> > are), it would help me to know when to stop working on the branch, and all
> > of the committers should probably sign up.
> > On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
> >
>
>
> Yes there are - the commits mailing list receives the notifications
> from both jenkins.cs.o and builds.a.o.
> Thanks for asking the question, I just assumed everyone knew of, and
> was subscribed to the mailing list.
>
> You can subscribe by sending an email to:
>
> cloudstack-commits-subscribe@incubator.apache.org
>
> --David
>
I've noticed that although builds.a.o is configured to email the
committers when the build breaks for their changes, it's sending to
weird addresses. For example, I see it sending notices to
chip.childers@apache.org, which doesn't exist. It seems to take the
username from the committer email address, and then use @a.o. Weird.
Also, perhaps we can configure the jobs to email build failures and "all
better" emails to the dev list instead of commits? I know that
committers should be watching the commits, but I bet that doesn't happen
in real time... and that the dev list would be a better place to get
attention a little more quickly.
-chip
Re: How we treat broken builds.
Posted by prasanna <ts...@apache.org>.
On 18 February 2013 09:01, David Nalley <da...@gnsa.us> wrote:
> On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com> wrote:
>> Are there jenkins reports the committers can subscribe to and get email
>> when there are breakages? I know at least once 4.1 was broken while I was
>> in the process of testing a commit (just a quick sanity check I do to avoid
>> being the guy who causes these sorts of things), and then I committed on
>> top of that broken branch. If there are such emails (and I suspect there
>> are), it would help me to know when to stop working on the branch, and all
>> of the committers should probably sign up.
>> On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
>>
>
>
> Yes there are - the commits mailing list receives the notifications
> from both jenkins.cs.o and builds.a.o.
> Thanks for asking the question, I just assumed everyone knew of, and
> was subscribed to the mailing list.
>
> You can subscribe by sending an email to:
>
> cloudstack-commits-subscribe@incubator.apache.org
>
> --David
Sorry - the apidoc broken is my fault. I pushed in a fix for running
marvin configs via mvn on Fri and emailed the lists. I mvn compiled
marvin but not the entire project. At the time a usage test was
failing. Was hoping to fix it once I got home but couldn't get any
time on the weekend.
Will fix this now. Sorry for the troubles.
Lesson learnt : not to push fixes before leaving work.
Re: How we treat broken builds.
Posted by David Nalley <da...@gnsa.us>.
On Sun, Feb 17, 2013 at 9:03 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> Are there jenkins reports the committers can subscribe to and get email
> when there are breakages? I know at least once 4.1 was broken while I was
> in the process of testing a commit (just a quick sanity check I do to avoid
> being the guy who causes these sorts of things), and then I committed on
> top of that broken branch. If there are such emails (and I suspect there
> are), it would help me to know when to stop working on the branch, and all
> of the committers should probably sign up.
> On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
>
Yes there are - the commits mailing list receives the notifications
from both jenkins.cs.o and builds.a.o.
Thanks for asking the question, I just assumed everyone knew of, and
was subscribed to the mailing list.
You can subscribe by sending an email to:
cloudstack-commits-subscribe@incubator.apache.org
--David
Re: How we treat broken builds.
Posted by Marcus Sorensen <sh...@gmail.com>.
Are there jenkins reports the committers can subscribe to and get email
when there are breakages? I know at least once 4.1 was broken while I was
in the process of testing a commit (just a quick sanity check I do to avoid
being the guy who causes these sorts of things), and then I committed on
top of that broken branch. If there are such emails (and I suspect there
are), it would help me to know when to stop working on the branch, and all
of the committers should probably sign up.
On Feb 17, 2013 6:44 PM, "David Nalley" <da...@gnsa.us> wrote:
Hi folks,
I know folks have said this elsewhere, but I'd like to reiterate it.
I am somewhat frustrated with our reaction to broken builds. In
general it seems we don't care, and this makes it more difficult to
fix problems. Jenkins reporting a broken build (be it a broken run of
RAT, failure to compile, failure of a unit test, building docs, etc.)
should be our Andon cord [1]. We should all stop commits that aren't
fixing the broken build. To illustrate why this is a problem,
RAT failures started occurring recently, this keeps us from testing
whether CloudStack builds, because each build is conditioned on the
successful completion of the test before it. That in turn keeps
apidocs from building, which keeps marvin from building, which keeps
documentation from building. We essentially are blind until it gets
fixed.
That means, that like TPS, when the andon cord gets pulled we all need
to focus on the problem, and not continuing our own work and ignoring
the problem and potentially contributing to making the solution more
painful.
My requests:
If you see that the branch you are working on is broken - please don't
commit to it, unless your commit is going to fix it.
If you see that the branch you are working on is broken - please help
fix it. (This should become priority #1, for everyone)
Please don't make a commit the last thing you do before going offline
- make sure that your commit isn't breaking any of the tests before
you leave. If your commit breaks something and you've gone offline for
16 hours, you've made life painful for others, so make sure there is
some buffer between your last commit and you going offline so you can
see and remedy any problems that arise.
And just for the record - as of this moment, 4.1 and master are broken
at various stages.
--David
[1] http://en.wikipedia.org/wiki/Andon_(manufacturing)