Release Process

[David Neuman <da...@gmail.com>]

I have been thinking about how we can get better with our release cadence.
I feel like we have slowed to a crawl and not been as good as we should
about how we release.  Our last Major release was in March and we haven't
had a real release since.   Moving forward I would like to see us get on a
more consistent cadence and provide smaller releases that can be more
easily digested by the community.  It's my hope that we can get to a
cadence where we are doing releases every 4 to 6 weeks,  we are releasing
in such a way that not all releases all required (e.g. if you are on 4.0
you don't need 4.1, you can just install 4.2), and we are more deliberate
about what we put into a release.

I think we can accomplish this by using our release branches better.  For
each major release we will cut a release branch from master (e.g. 4.x) and
then we will use cherry-picking to add new features and bug fixes to that
release.  This means that if 4.0 has been released and you want to get your
feature/bug fix in 4.1, you will first submit your PR to master and then
cheery pick your squash merged commit to the 4.x branch which we will use
to create the 4.1 release.  I think we should allow either the contributor
or the committer (who is merging the PR) to suggest if a feature goes into
the release branch, and if we disagree we can take it to the list.  If we
decide that every PR to master also goes into 4.1 then so be it, but at
least we are making a conscious decision of what goes into the next
release.   This will allow us to not be so worried about what we are
merging into master and how that will affect our next release.  According
to our 6 week cadence we will cut a new release from the current feature
branch and put that up for a vote.  These releases will be small and
testable enough that we can get feedback quickly and move from RC to
release in a short time.  If a release has too many issues then we may end
up deciding to skip that minor release in favor of the next one, hopefully
this does not happen very often if at all.

Once a breaking change is introduced to master which will require a new
major release, we will create a new major release branch from master (which
will mean a new release manager).  We will then repeat the same process
with the new release branch, etc, etc.

As for LTS we will provide support for the latest major release plus the
one previous.  So, once we release 4.0 we will support 4.0 and 3.1.  Any
security issues that arise will -- if present -- be applied to each of
these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc, etc.

Please let me know if you have any thoughts on this plan.  If there are no
major objections I would like to try this with 4.x with the idea that we
will adjust how we do things as necessary.

We need to get better at releasing and something needs to change, hopefully
this can get us going in the right direction.

Thanks,
Dave


TL;DR -  I am proposing a 6 week release cycle using cherry-picks to our
release branch to control what goes into a minor release.  Major releases
will be cut from master as necessary - such as if there is a breaking
change that is introduced.

Re: [EXTERNAL] Re: Release Process

[Jeremy Mitchell <mi...@gmail.com>]

Ok, I'm starting to see the light. If we moved faster, 4.1, 4.2, 4.3 for
example 6 weeks apart then maybe where I work (Comcast) would get on board
with actually using those releases (rather than some arbitrary point on
master because the open source releases are moving too slow for us).

But I still think cherry-picking select features into a release could be
problematic. I would suggest:

4.0 (from master)
4.1 (from master)
4.2 (from master)
...
5.0 (from master) and the only reason for the major bump is something
pretty big went into master.

also +100 on CI that:

1. actually runs...consistently (I think we should look more into these
Github Actions)
2. tests a good deal of things (build, unit tests, api tests, UI tests,
maybe some integration tests?)

jeremy

On Wed, Oct 30, 2019 at 9:56 AM ocket 8888 <oc...@gmail.com> wrote:

> >   What I was really getting at is if ANY company has a recent version in
> a
> >   test or prod-like enviro (meaning the version is being exercised and
> >   thoroughly tested), then maybe we consider simply cutting a release
> from
> >   that version
>
> I think the idea s that companies should stop doing their own versioning -
> which was totally necessary because it's unreasonable to wait half a year
> for an Apache release - and start deploying official releases. That way the
> releases ARE thoroughly exercised and tested, just that the commit hashes
> aren't dictated by the companies. Companies could refuse, of course, but
> that only makes things harder for them IMO. At Comcast it's taken
> notoriously long to validate releases, and making them smaller will help
> that.
>
> > In my opinion, in order to get to a cadence we are discussing we need to
> put a lot more work into the CI system.
>
> I agree, it's all but worthless at the moment and testing is vital to
> providing stable releases. I know GitHub Actions have been discussed
> before, but they are coming out of beta in a couple of weeks and if the
> problem with CI is capacity then maybe we could disable the Jenkins jobs
> and set up Actions in their stead.
>
> On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> wrote:
>
> > In my opinion, in order to get to a cadence we are discussing we need to
> > put a lot more work into the CI system. It has been failing consistently,
> > doesn’t block PR approvals/merges and when it actually runs it does not
> > test anything besides build and license headers.
> >
> > In the past couple of weeks we have had (off the top of my head):
> >
> > * Unit tests broken
> > * TO API tests broken
> > * Golang vendor issues
> > * TO Go build issues
> >
> > All were merged and broke master.
> >
> > Yes, having release branches sounds great and more aggressive cadence
> will
> > minimize the amount of risk but I think that comes with the need to
> improve
> > on our automated validation and testing.
> >
> > We have proven that we have not kept master in a good state and adding
> > more releases (more branches) will make it even harder to keep that
> > stability we are already fighting.
> >
> > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
> >
> >     Yeah, I get it. No one company should be driving release
> > schedules/scope.
> >
> >     What I was really getting at is if ANY company has a recent version
> in
> > a
> >     test or prod-like enviro (meaning the version is being exercised and
> >     thoroughly tested), then maybe we consider simply cutting a release
> > from
> >     that version. For example, imagine company X had this version in a
> >     test/prod enviro:
> >
> >     Master-10287.7e62d07 (
> >
> >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > )
> >
> >     Then I would be in favor of cutting the 4.0 release from
> >     Master-10287.7e62d07 as we know it is proven to work based on company
> > X's
> >     actual use of it.
> >
> >     The truth is (imo), we don't have the bandwidth to manually verify
> > releases
> >     that we are not using/have not used nor do we have the necessary
> > automated
> >     test coverage to verify these releases. So most of our releases are
> >     significantly untested.
> >
> >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> > wrote:
> >
> >     > I'm really not a fan of allowing Comcast to dictate the release
> > scope and
> >     > schedule. If cherry-picking is too messy, then we can just cut new
> > minor
> >     > releases directly from master.
> >     >
> >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mitchell852@gmail.com
> >
> > wrote:
> >     >
> >     > > I don't think it's as easy as cherry picking (backporting)
> certain
> >     > features
> >     > > into a release branch. I could be wrong but I really don't think
> > it is.
> >     > >
> >     > > So what I'm hearing is that 4.0 gets cut from master and we go
> > through
> >     > with
> >     > > our normal process of testing, validating, etc. At the same time,
> > 4.1
> >     > > branch is created from 4.0. Master moves on as normal. Let's say
> 25
> >     > commits
> >     > > come in to master during the next 6 weeks. During that 6 weeks,
> we
> > cherry
> >     > > pick only the features that look interesting for 4.1? (the rest
> > just stay
> >     > > in master and will get picked up in 5.0?)
> >     > >
> >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks
> > apart
> >     > and
> >     > > have only "interesting" features so the releases are
> incrementally
> > very
> >     > > small and much easier to test/validate, however, those features
> > might
> >     > > depend on other commits. At some point, cherry picking becomes
> > impossible
> >     > > as the foundation of the code base has shifted so drastically.
> > Cherry
> >     > > picking will become very messy and introduce a high level of risk
> > imo.
> >     > >
> >     > > Also, you'd have to manage the scope of these minor releases
> which
> > we are
> >     > > not very good at because the designated release manager has a
> > full-time
> >     > job
> >     > > to attend to and now we have even more releases to
> >     > > scope/test/validate/approve. I like the idea of faster releases
> > but this
> >     > > sounds like more work that nobody has the bandwidth for.
> >     > >
> >     > > Ok, so rather than shooting down an idea with no alternative
> > proposal, I
> >     > > propose this. At Comcast, we are trying to get to the point where
> > we
> >     > > release TC to prod from the head of master every few weeks. What
> > if every
> >     > > time Comcast does an internal upgrade, they note the commit hash
> > and once
> >     > > comcast is happy with the upgrade in their prod enviro, they
> > propose a
> >     > > release pinned to that commit hash? With this approach you'd end
> > up with:
> >     > >
> >     > > 1. fast releases (hopefully every few weeks)
> >     > > 2. well tested releases as it's in a real prod enviro
> >     > > 3. little/no validation work for anyone
> >     > >
> >     > > Also, this doesn't have to be Comcast. If anyone else is out
> front
> > at the
> >     > > edge of master and running it in a trusted enviro, they could
> > propose a
> >     > > release as well.
> >     > >
> >     > > Jeremy
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > david.neuman64@gmail.com>
> >     > > wrote:
> >     > >
> >     > > > I have been thinking about how we can get better with our
> release
> >     > > cadence.
> >     > > > I feel like we have slowed to a crawl and not been as good as
> we
> > should
> >     > > > about how we release.  Our last Major release was in March and
> we
> >     > haven't
> >     > > > had a real release since.   Moving forward I would like to see
> > us get
> >     > on
> >     > > a
> >     > > > more consistent cadence and provide smaller releases that can
> be
> > more
> >     > > > easily digested by the community.  It's my hope that we can get
> > to a
> >     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
> >     > releasing
> >     > > > in such a way that not all releases all required (e.g. if you
> > are on
> >     > 4.0
> >     > > > you don't need 4.1, you can just install 4.2), and we are more
> >     > deliberate
> >     > > > about what we put into a release.
> >     > > >
> >     > > > I think we can accomplish this by using our release branches
> > better.
> >     > For
> >     > > > each major release we will cut a release branch from master
> > (e.g. 4.x)
> >     > > and
> >     > > > then we will use cherry-picking to add new features and bug
> > fixes to
> >     > that
> >     > > > release.  This means that if 4.0 has been released and you want
> > to get
> >     > > your
> >     > > > feature/bug fix in 4.1, you will first submit your PR to master
> > and
> >     > then
> >     > > > cheery pick your squash merged commit to the 4.x branch which
> we
> > will
> >     > use
> >     > > > to create the 4.1 release.  I think we should allow either the
> >     > > contributor
> >     > > > or the committer (who is merging the PR) to suggest if a
> feature
> > goes
> >     > > into
> >     > > > the release branch, and if we disagree we can take it to the
> > list.  If
> >     > we
> >     > > > decide that every PR to master also goes into 4.1 then so be
> it,
> > but at
> >     > > > least we are making a conscious decision of what goes into the
> > next
> >     > > > release.   This will allow us to not be so worried about what
> we
> > are
> >     > > > merging into master and how that will affect our next release.
> >     > According
> >     > > > to our 6 week cadence we will cut a new release from the
> current
> >     > feature
> >     > > > branch and put that up for a vote.  These releases will be
> small
> > and
> >     > > > testable enough that we can get feedback quickly and move from
> > RC to
> >     > > > release in a short time.  If a release has too many issues then
> > we may
> >     > > end
> >     > > > up deciding to skip that minor release in favor of the next
> one,
> >     > > hopefully
> >     > > > this does not happen very often if at all.
> >     > > >
> >     > > > Once a breaking change is introduced to master which will
> > require a new
> >     > > > major release, we will create a new major release branch from
> > master
> >     > > (which
> >     > > > will mean a new release manager).  We will then repeat the same
> > process
> >     > > > with the new release branch, etc, etc.
> >     > > >
> >     > > > As for LTS we will provide support for the latest major release
> > plus
> >     > the
> >     > > > one previous.  So, once we release 4.0 we will support 4.0 and
> > 3.1.
> >     > Any
> >     > > > security issues that arise will -- if present -- be applied to
> > each of
> >     > > > these versions.  Once 4.1 is released we will support 4.1 and
> > 3.1, etc,
> >     > > > etc.
> >     > > >
> >     > > > Please let me know if you have any thoughts on this plan.  If
> > there are
> >     > > no
> >     > > > major objections I would like to try this with 4.x with the
> idea
> > that
> >     > we
> >     > > > will adjust how we do things as necessary.
> >     > > >
> >     > > > We need to get better at releasing and something needs to
> change,
> >     > > hopefully
> >     > > > this can get us going in the right direction.
> >     > > >
> >     > > > Thanks,
> >     > > > Dave
> >     > > >
> >     > > >
> >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > cherry-picks to
> >     > our
> >     > > > release branch to control what goes into a minor release.
> Major
> >     > releases
> >     > > > will be cut from master as necessary - such as if there is a
> > breaking
> >     > > > change that is introduced.
> >     > > >
> >     > >
> >     >
> >
> >
> >
>

Re: [EXTERNAL] Re: Release Process

[ocket 8888 <oc...@gmail.com>]

>   What I was really getting at is if ANY company has a recent version in a
>   test or prod-like enviro (meaning the version is being exercised and
>   thoroughly tested), then maybe we consider simply cutting a release from
>   that version

I think the idea s that companies should stop doing their own versioning -
which was totally necessary because it's unreasonable to wait half a year
for an Apache release - and start deploying official releases. That way the
releases ARE thoroughly exercised and tested, just that the commit hashes
aren't dictated by the companies. Companies could refuse, of course, but
that only makes things harder for them IMO. At Comcast it's taken
notoriously long to validate releases, and making them smaller will help
that.

> In my opinion, in order to get to a cadence we are discussing we need to
put a lot more work into the CI system.

I agree, it's all but worthless at the moment and testing is vital to
providing stable releases. I know GitHub Actions have been discussed
before, but they are coming out of beta in a couple of weeks and if the
problem with CI is capacity then maybe we could disable the Jenkins jobs
and set up Actions in their stead.

On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael <Mi...@comcast.com>
wrote:

> In my opinion, in order to get to a cadence we are discussing we need to
> put a lot more work into the CI system. It has been failing consistently,
> doesn’t block PR approvals/merges and when it actually runs it does not
> test anything besides build and license headers.
>
> In the past couple of weeks we have had (off the top of my head):
>
> * Unit tests broken
> * TO API tests broken
> * Golang vendor issues
> * TO Go build issues
>
> All were merged and broke master.
>
> Yes, having release branches sounds great and more aggressive cadence will
> minimize the amount of risk but I think that comes with the need to improve
> on our automated validation and testing.
>
> We have proven that we have not kept master in a good state and adding
> more releases (more branches) will make it even harder to keep that
> stability we are already fighting.
>
> On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
>
>     Yeah, I get it. No one company should be driving release
> schedules/scope.
>
>     What I was really getting at is if ANY company has a recent version in
> a
>     test or prod-like enviro (meaning the version is being exercised and
>     thoroughly tested), then maybe we consider simply cutting a release
> from
>     that version. For example, imagine company X had this version in a
>     test/prod enviro:
>
>     Master-10287.7e62d07 (
>
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> )
>
>     Then I would be in favor of cutting the 4.0 release from
>     Master-10287.7e62d07 as we know it is proven to work based on company
> X's
>     actual use of it.
>
>     The truth is (imo), we don't have the bandwidth to manually verify
> releases
>     that we are not using/have not used nor do we have the necessary
> automated
>     test coverage to verify these releases. So most of our releases are
>     significantly untested.
>
>     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> wrote:
>
>     > I'm really not a fan of allowing Comcast to dictate the release
> scope and
>     > schedule. If cherry-picking is too messy, then we can just cut new
> minor
>     > releases directly from master.
>     >
>     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com>
> wrote:
>     >
>     > > I don't think it's as easy as cherry picking (backporting) certain
>     > features
>     > > into a release branch. I could be wrong but I really don't think
> it is.
>     > >
>     > > So what I'm hearing is that 4.0 gets cut from master and we go
> through
>     > with
>     > > our normal process of testing, validating, etc. At the same time,
> 4.1
>     > > branch is created from 4.0. Master moves on as normal. Let's say 25
>     > commits
>     > > come in to master during the next 6 weeks. During that 6 weeks, we
> cherry
>     > > pick only the features that look interesting for 4.1? (the rest
> just stay
>     > > in master and will get picked up in 5.0?)
>     > >
>     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks
> apart
>     > and
>     > > have only "interesting" features so the releases are incrementally
> very
>     > > small and much easier to test/validate, however, those features
> might
>     > > depend on other commits. At some point, cherry picking becomes
> impossible
>     > > as the foundation of the code base has shifted so drastically.
> Cherry
>     > > picking will become very messy and introduce a high level of risk
> imo.
>     > >
>     > > Also, you'd have to manage the scope of these minor releases which
> we are
>     > > not very good at because the designated release manager has a
> full-time
>     > job
>     > > to attend to and now we have even more releases to
>     > > scope/test/validate/approve. I like the idea of faster releases
> but this
>     > > sounds like more work that nobody has the bandwidth for.
>     > >
>     > > Ok, so rather than shooting down an idea with no alternative
> proposal, I
>     > > propose this. At Comcast, we are trying to get to the point where
> we
>     > > release TC to prod from the head of master every few weeks. What
> if every
>     > > time Comcast does an internal upgrade, they note the commit hash
> and once
>     > > comcast is happy with the upgrade in their prod enviro, they
> propose a
>     > > release pinned to that commit hash? With this approach you'd end
> up with:
>     > >
>     > > 1. fast releases (hopefully every few weeks)
>     > > 2. well tested releases as it's in a real prod enviro
>     > > 3. little/no validation work for anyone
>     > >
>     > > Also, this doesn't have to be Comcast. If anyone else is out front
> at the
>     > > edge of master and running it in a trusted enviro, they could
> propose a
>     > > release as well.
>     > >
>     > > Jeremy
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> david.neuman64@gmail.com>
>     > > wrote:
>     > >
>     > > > I have been thinking about how we can get better with our release
>     > > cadence.
>     > > > I feel like we have slowed to a crawl and not been as good as we
> should
>     > > > about how we release.  Our last Major release was in March and we
>     > haven't
>     > > > had a real release since.   Moving forward I would like to see
> us get
>     > on
>     > > a
>     > > > more consistent cadence and provide smaller releases that can be
> more
>     > > > easily digested by the community.  It's my hope that we can get
> to a
>     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
>     > releasing
>     > > > in such a way that not all releases all required (e.g. if you
> are on
>     > 4.0
>     > > > you don't need 4.1, you can just install 4.2), and we are more
>     > deliberate
>     > > > about what we put into a release.
>     > > >
>     > > > I think we can accomplish this by using our release branches
> better.
>     > For
>     > > > each major release we will cut a release branch from master
> (e.g. 4.x)
>     > > and
>     > > > then we will use cherry-picking to add new features and bug
> fixes to
>     > that
>     > > > release.  This means that if 4.0 has been released and you want
> to get
>     > > your
>     > > > feature/bug fix in 4.1, you will first submit your PR to master
> and
>     > then
>     > > > cheery pick your squash merged commit to the 4.x branch which we
> will
>     > use
>     > > > to create the 4.1 release.  I think we should allow either the
>     > > contributor
>     > > > or the committer (who is merging the PR) to suggest if a feature
> goes
>     > > into
>     > > > the release branch, and if we disagree we can take it to the
> list.  If
>     > we
>     > > > decide that every PR to master also goes into 4.1 then so be it,
> but at
>     > > > least we are making a conscious decision of what goes into the
> next
>     > > > release.   This will allow us to not be so worried about what we
> are
>     > > > merging into master and how that will affect our next release.
>     > According
>     > > > to our 6 week cadence we will cut a new release from the current
>     > feature
>     > > > branch and put that up for a vote.  These releases will be small
> and
>     > > > testable enough that we can get feedback quickly and move from
> RC to
>     > > > release in a short time.  If a release has too many issues then
> we may
>     > > end
>     > > > up deciding to skip that minor release in favor of the next one,
>     > > hopefully
>     > > > this does not happen very often if at all.
>     > > >
>     > > > Once a breaking change is introduced to master which will
> require a new
>     > > > major release, we will create a new major release branch from
> master
>     > > (which
>     > > > will mean a new release manager).  We will then repeat the same
> process
>     > > > with the new release branch, etc, etc.
>     > > >
>     > > > As for LTS we will provide support for the latest major release
> plus
>     > the
>     > > > one previous.  So, once we release 4.0 we will support 4.0 and
> 3.1.
>     > Any
>     > > > security issues that arise will -- if present -- be applied to
> each of
>     > > > these versions.  Once 4.1 is released we will support 4.1 and
> 3.1, etc,
>     > > > etc.
>     > > >
>     > > > Please let me know if you have any thoughts on this plan.  If
> there are
>     > > no
>     > > > major objections I would like to try this with 4.x with the idea
> that
>     > we
>     > > > will adjust how we do things as necessary.
>     > > >
>     > > > We need to get better at releasing and something needs to change,
>     > > hopefully
>     > > > this can get us going in the right direction.
>     > > >
>     > > > Thanks,
>     > > > Dave
>     > > >
>     > > >
>     > > > TL;DR -  I am proposing a 6 week release cycle using
> cherry-picks to
>     > our
>     > > > release branch to control what goes into a minor release.  Major
>     > releases
>     > > > will be cut from master as necessary - such as if there is a
> breaking
>     > > > change that is introduced.
>     > > >
>     > >
>     >
>
>
>

Re: [EXTERNAL] Re: Release Process

[David Neuman <da...@gmail.com>]

No, that's not correct.
You have a 4.x branch that almost everything gets cherry-picked to.  Think
of this as now being the "stable" branch.  Everything gets committed to
master and anything we want to release gets cherry picked into the stable
branch, 4.x.  This should be everything that is complete and ready to go,
we shouldn't be picking and choosing.  We should have to find a reason NOT
to include something rather than a reason TO include something.
When you are ready for a release (e.g. after 6 weeks) you create 4.0.x,
4.1.x,  off of 4.x and you will not cherry-pick anything other than
bug-fixes that affect the release into these branches once they are
created.

This allows us to not have to force master to be stable while still having
a stable/trusted branch.  We should still be running our automated testing
against master and release branches on each PR/commit as well as every
night (again a different discussion).




On Wed, Oct 30, 2019 at 3:28 PM Jeremy Mitchell <mi...@gmail.com>
wrote:

> still -1 on on cherry-picked releases if this is what i'm hearing:
>
> 4.0 (derived from master)
> 4.1 (derived from 4.0 plus some cherry picks from master introduced during
> the last 6 weeks)
> 4.2 (derived from 4.1 plus some cherry picks from master introduced during
> the last 6 weeks)
> 4.3 (derived from 4.2 plus some cherry picks from master introduced during
> the last 6 weeks)
>
> ^^ 4.3 is a triple cherry-picked release. i would not trust that at all.
>
> in theory it might sound great but in practice i think it will cause
> problems like rawlin said (risky, tedious, and
> represents unnecessary effort that would be better spent elsewhere)
>
> jeremy
>
> On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <Jo...@comcast.com>
> wrote:
>
> > Did we define why/how what we've been doing isnt' working?  I see a
> > consensus that it's not, suggestions on alternatives, and an ambition for
> > faster and more stable; but not a definition of the thing is the reason
> why
> > we must change?    Not saying we shouldn't, but I'm just asking so we try
> > not to repeat the same mistakes following new steps.  Usually I've found
> > that release blockage is derived from not defining clear objectives and
> > definitions of good enough (not necessarily perfect) consistently
> > throughout a release cycle.  It's not enough to start a release cycle
> with
> > one scope of both, yet at the end continue to revise that definition.
> >
> > Jonathan G
> >
> >
> > On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:
> >
> >     Sorry, I have taken too long to respond.
> >
> >     +1000000 on better CI and Testing, I think we all agree that needs to
> > get
> >     better and is a different beast than this topic.
> >
> >     As for the release branches, I am trying to follow a process similar
> to
> >     ATS, which is usually the model we try to follow with our project.
> >     While I agree that branching from master is the ideal solution, we
> > tried
> >     that and it didn't work.  I feel like we are trying to agree to do
> the
> > same
> >     thing that we already tried and didn't work.  I think we should have
> a
> >     little freedom to commit things into master and, if they break
> things,
> > work
> >     on getting them fixed.  We also might commit PRs to master that are
> > partial
> >     implementations of a bigger feature.   If we are using a time based
> >     strategy to cut our releases from master then we are going to be
> >     continually worrying about making sure Master stays not broken.  I
> > think we
> >     should try to keep master deployable, but it's fine if it's not for
> > short
> >     periods of time.
> >
> >     I am not trying to suggest that we are "picky" about what goes into
> our
> >     release branch.  I think we should try to cherry-pick as much as we
> can
> >     into our release branch once it is ready to be cherry-picked.  I
> think
> > we
> >     should be able to take the tip of our release branch and deploy it
> > whenever
> >     we want with some confidence since we know that features/bug-fixes
> that
> >     have been cherry-picked have already been reviewed and tested against
> >     master.  Yes, its more work, but it also give us the ability to have
> > more
> >     confidence in what is in our release at any given time.  Again, we
> > should
> >     be trying to cherry-pick in everything that makes sense to
> cherry-pick.
> >
> >     I think the process that I would like to see looks like this:
> >     - Cut a 4.x branch from Master
> >     - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > branch
> >     - Tag the 4.0.0 release from the 4.0.x branch
> >     - New non-breaking changes and bug fixes that go into master are also
> >     Cherry Picked to 4.x
> >     - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when
> > we
> >     are ready for a RC.
> >     - Rinse and repeat for all 4.x releases
> >     - Once there's a need for a major release we make a 5.x branch from
> > master
> >
> >     If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > then
> >     4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > have
> >     4.1 yet).  Yes, that is a more work than today, but I think A) it's
> not
> >     really that bad and B) it gives us more control.
> >
> >     This should allow us to deploy much quicker and with more confidence
> > than
> >     is possible today.   We can also choose to deploy from any point in
> > time
> >     from the 4.x branch with relatively high confidence.
> >
> >     I'll wrap this up by saying that I don't think there's a perfect
> > solution,
> >     but I think we can agree on something that is worth trying at least
> > with
> >     the 4.x release life cycle.  Once we have some experience, we can
> > figure
> >     out where it is not meeting our needs and figure out how to adjust as
> >     necessary.  I think we can all agree our current process is not
> > working.
> >
> >     Thanks,
> >     Dave
> >
> >
> >
> >     On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > wrote:
> >
> >     > +1 on more frequent releases
> >     > +1 on improving our CI. We need to get our CI to the point where
> > we're
> >     > confident deploying changes to production if the CI passes.
> >     > +1 on "trunk-based" development as opposed to GitFlow
> >     > -1 on cherry-picking new features. This is risky, tedious, and
> >     > represents unnecessary effort that would be better spent elsewhere,
> >     > like improving our CI or adding more automated tests.
> >     > +1 on cutting minor release branches directly off master
> >     >
> >     > I think cutting minor release branches directly off master can put
> us
> >     > in a win-win scenario. Our releases become smaller and more
> frequent,
> >     > and we avoid the unnecessary risk and effort of cherry-picking new
> >     > features onto stale release branches.
> >     >
> >     > This is the process that I would propose:
> >     > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> >     > 2. Develop master for 4-6 weeks.
> >     > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> >     > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> >     > 5. Whenever a major (show-stopping) bug or security issue is found,
> >     > cherry-pick the fix from master into the latest release branch
> (e.g.
> >     > 4.1.x), and create a 4.1.1 RC.
> >     >
> >     > As the 4.x release manager, I really don't want to get into the
> >     > business of selecting what should and should not go into a
> particular
> >     > release. Every 4-6 weeks, we should just cut a new minor release
> >     > branch off master and take everything that was contributed during
> > that
> >     > time, as long as everything builds fine and all the tests pass. The
> >     > master branch, while it doesn't always have to be perfect, should
> at
> >     > least always be deployable. If a new feature is merged with bugs
> and
> >     > becomes a show-stopper, it should be either immediately remedied or
> >     > reverted on master while we work on a fix.
> >     >
> >     > - Rawlin
> >     >
> >     > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> >     > <Mi...@comcast.com> wrote:
> >     > >
> >     > > In my opinion, in order to get to a cadence we are discussing we
> > need to
> >     > put a lot more work into the CI system. It has been failing
> > consistently,
> >     > doesn’t block PR approvals/merges and when it actually runs it does
> > not
> >     > test anything besides build and license headers.
> >     > >
> >     > > In the past couple of weeks we have had (off the top of my head):
> >     > >
> >     > > * Unit tests broken
> >     > > * TO API tests broken
> >     > > * Golang vendor issues
> >     > > * TO Go build issues
> >     > >
> >     > > All were merged and broke master.
> >     > >
> >     > > Yes, having release branches sounds great and more aggressive
> > cadence
> >     > will minimize the amount of risk but I think that comes with the
> > need to
> >     > improve on our automated validation and testing.
> >     > >
> >     > > We have proven that we have not kept master in a good state and
> > adding
> >     > more releases (more branches) will make it even harder to keep that
> >     > stability we are already fighting.
> >     > >
> >     > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> > wrote:
> >     > >
> >     > >     Yeah, I get it. No one company should be driving release
> >     > schedules/scope.
> >     > >
> >     > >     What I was really getting at is if ANY company has a recent
> > version
> >     > in a
> >     > >     test or prod-like enviro (meaning the version is being
> > exercised and
> >     > >     thoroughly tested), then maybe we consider simply cutting a
> > release
> >     > from
> >     > >     that version. For example, imagine company X had this version
> > in a
> >     > >     test/prod enviro:
> >     > >
> >     > >     Master-10287.7e62d07 (
> >     > >
> >     >
> >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> >     > )
> >     > >
> >     > >     Then I would be in favor of cutting the 4.0 release from
> >     > >     Master-10287.7e62d07 as we know it is proven to work based on
> >     > company X's
> >     > >     actual use of it.
> >     > >
> >     > >     The truth is (imo), we don't have the bandwidth to manually
> > verify
> >     > releases
> >     > >     that we are not using/have not used nor do we have the
> > necessary
> >     > automated
> >     > >     test coverage to verify these releases. So most of our
> > releases are
> >     > >     significantly untested.
> >     > >
> >     > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> > ocket8888@gmail.com>
> >     > wrote:
> >     > >
> >     > >     > I'm really not a fan of allowing Comcast to dictate the
> > release
> >     > scope and
> >     > >     > schedule. If cherry-picking is too messy, then we can just
> > cut new
> >     > minor
> >     > >     > releases directly from master.
> >     > >     >
> >     > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > mitchell852@gmail.com>
> >     > wrote:
> >     > >     >
> >     > >     > > I don't think it's as easy as cherry picking
> (backporting)
> >     > certain
> >     > >     > features
> >     > >     > > into a release branch. I could be wrong but I really
> don't
> > think
> >     > it is.
> >     > >     > >
> >     > >     > > So what I'm hearing is that 4.0 gets cut from master and
> > we go
> >     > through
> >     > >     > with
> >     > >     > > our normal process of testing, validating, etc. At the
> same
> >     > time, 4.1
> >     > >     > > branch is created from 4.0. Master moves on as normal.
> > Let's say
> >     > 25
> >     > >     > commits
> >     > >     > > come in to master during the next 6 weeks. During that 6
> > weeks,
> >     > we cherry
> >     > >     > > pick only the features that look interesting for 4.1?
> (the
> > rest
> >     > just stay
> >     > >     > > in master and will get picked up in 5.0?)
> >     > >     > >
> >     > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> roughly 6
> >     > weeks apart
> >     > >     > and
> >     > >     > > have only "interesting" features so the releases are
> >     > incrementally very
> >     > >     > > small and much easier to test/validate, however, those
> > features
> >     > might
> >     > >     > > depend on other commits. At some point, cherry picking
> > becomes
> >     > impossible
> >     > >     > > as the foundation of the code base has shifted so
> > drastically.
> >     > Cherry
> >     > >     > > picking will become very messy and introduce a high level
> > of
> >     > risk imo.
> >     > >     > >
> >     > >     > > Also, you'd have to manage the scope of these minor
> > releases
> >     > which we are
> >     > >     > > not very good at because the designated release manager
> > has a
> >     > full-time
> >     > >     > job
> >     > >     > > to attend to and now we have even more releases to
> >     > >     > > scope/test/validate/approve. I like the idea of faster
> > releases
> >     > but this
> >     > >     > > sounds like more work that nobody has the bandwidth for.
> >     > >     > >
> >     > >     > > Ok, so rather than shooting down an idea with no
> > alternative
> >     > proposal, I
> >     > >     > > propose this. At Comcast, we are trying to get to the
> point
> >     > where we
> >     > >     > > release TC to prod from the head of master every few
> > weeks. What
> >     > if every
> >     > >     > > time Comcast does an internal upgrade, they note the
> > commit hash
> >     > and once
> >     > >     > > comcast is happy with the upgrade in their prod enviro,
> > they
> >     > propose a
> >     > >     > > release pinned to that commit hash? With this approach
> > you'd end
> >     > up with:
> >     > >     > >
> >     > >     > > 1. fast releases (hopefully every few weeks)
> >     > >     > > 2. well tested releases as it's in a real prod enviro
> >     > >     > > 3. little/no validation work for anyone
> >     > >     > >
> >     > >     > > Also, this doesn't have to be Comcast. If anyone else is
> > out
> >     > front at the
> >     > >     > > edge of master and running it in a trusted enviro, they
> > could
> >     > propose a
> >     > >     > > release as well.
> >     > >     > >
> >     > >     > > Jeremy
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> >     > david.neuman64@gmail.com>
> >     > >     > > wrote:
> >     > >     > >
> >     > >     > > > I have been thinking about how we can get better with
> our
> >     > release
> >     > >     > > cadence.
> >     > >     > > > I feel like we have slowed to a crawl and not been as
> > good as
> >     > we should
> >     > >     > > > about how we release.  Our last Major release was in
> > March and
> >     > we
> >     > >     > haven't
> >     > >     > > > had a real release since.   Moving forward I would like
> > to see
> >     > us get
> >     > >     > on
> >     > >     > > a
> >     > >     > > > more consistent cadence and provide smaller releases
> > that can
> >     > be more
> >     > >     > > > easily digested by the community.  It's my hope that we
> > can
> >     > get to a
> >     > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> > we are
> >     > >     > releasing
> >     > >     > > > in such a way that not all releases all required (e.g.
> > if you
> >     > are on
> >     > >     > 4.0
> >     > >     > > > you don't need 4.1, you can just install 4.2), and we
> > are more
> >     > >     > deliberate
> >     > >     > > > about what we put into a release.
> >     > >     > > >
> >     > >     > > > I think we can accomplish this by using our release
> > branches
> >     > better.
> >     > >     > For
> >     > >     > > > each major release we will cut a release branch from
> > master
> >     > (e.g. 4.x)
> >     > >     > > and
> >     > >     > > > then we will use cherry-picking to add new features and
> > bug
> >     > fixes to
> >     > >     > that
> >     > >     > > > release.  This means that if 4.0 has been released and
> > you
> >     > want to get
> >     > >     > > your
> >     > >     > > > feature/bug fix in 4.1, you will first submit your PR
> to
> >     > master and
> >     > >     > then
> >     > >     > > > cheery pick your squash merged commit to the 4.x branch
> > which
> >     > we will
> >     > >     > use
> >     > >     > > > to create the 4.1 release.  I think we should allow
> > either the
> >     > >     > > contributor
> >     > >     > > > or the committer (who is merging the PR) to suggest if
> a
> >     > feature goes
> >     > >     > > into
> >     > >     > > > the release branch, and if we disagree we can take it
> to
> > the
> >     > list.  If
> >     > >     > we
> >     > >     > > > decide that every PR to master also goes into 4.1 then
> > so be
> >     > it, but at
> >     > >     > > > least we are making a conscious decision of what goes
> > into the
> >     > next
> >     > >     > > > release.   This will allow us to not be so worried
> about
> > what
> >     > we are
> >     > >     > > > merging into master and how that will affect our next
> > release.
> >     > >     > According
> >     > >     > > > to our 6 week cadence we will cut a new release from
> the
> >     > current
> >     > >     > feature
> >     > >     > > > branch and put that up for a vote.  These releases will
> > be
> >     > small and
> >     > >     > > > testable enough that we can get feedback quickly and
> > move from
> >     > RC to
> >     > >     > > > release in a short time.  If a release has too many
> > issues
> >     > then we may
> >     > >     > > end
> >     > >     > > > up deciding to skip that minor release in favor of the
> > next
> >     > one,
> >     > >     > > hopefully
> >     > >     > > > this does not happen very often if at all.
> >     > >     > > >
> >     > >     > > > Once a breaking change is introduced to master which
> will
> >     > require a new
> >     > >     > > > major release, we will create a new major release
> branch
> > from
> >     > master
> >     > >     > > (which
> >     > >     > > > will mean a new release manager).  We will then repeat
> > the
> >     > same process
> >     > >     > > > with the new release branch, etc, etc.
> >     > >     > > >
> >     > >     > > > As for LTS we will provide support for the latest major
> >     > release plus
> >     > >     > the
> >     > >     > > > one previous.  So, once we release 4.0 we will support
> > 4.0 and
> >     > 3.1.
> >     > >     > Any
> >     > >     > > > security issues that arise will -- if present -- be
> > applied to
> >     > each of
> >     > >     > > > these versions.  Once 4.1 is released we will support
> > 4.1 and
> >     > 3.1, etc,
> >     > >     > > > etc.
> >     > >     > > >
> >     > >     > > > Please let me know if you have any thoughts on this
> > plan.  If
> >     > there are
> >     > >     > > no
> >     > >     > > > major objections I would like to try this with 4.x with
> > the
> >     > idea that
> >     > >     > we
> >     > >     > > > will adjust how we do things as necessary.
> >     > >     > > >
> >     > >     > > > We need to get better at releasing and something needs
> to
> >     > change,
> >     > >     > > hopefully
> >     > >     > > > this can get us going in the right direction.
> >     > >     > > >
> >     > >     > > > Thanks,
> >     > >     > > > Dave
> >     > >     > > >
> >     > >     > > >
> >     > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> >     > cherry-picks to
> >     > >     > our
> >     > >     > > > release branch to control what goes into a minor
> release.
> >     > Major
> >     > >     > releases
> >     > >     > > > will be cut from master as necessary - such as if there
> > is a
> >     > breaking
> >     > >     > > > change that is introduced.
> >     > >     > > >
> >     > >     > >
> >     > >     >
> >     > >
> >     > >
> >     >
> >
> >
> >
>

Re: [EXTERNAL] Re: Release Process

[Jeremy Mitchell <mi...@gmail.com>]

still -1 on on cherry-picked releases if this is what i'm hearing:

4.0 (derived from master)
4.1 (derived from 4.0 plus some cherry picks from master introduced during
the last 6 weeks)
4.2 (derived from 4.1 plus some cherry picks from master introduced during
the last 6 weeks)
4.3 (derived from 4.2 plus some cherry picks from master introduced during
the last 6 weeks)

^^ 4.3 is a triple cherry-picked release. i would not trust that at all.

in theory it might sound great but in practice i think it will cause
problems like rawlin said (risky, tedious, and
represents unnecessary effort that would be better spent elsewhere)

jeremy

On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <Jo...@comcast.com>
wrote:

> Did we define why/how what we've been doing isnt' working?  I see a
> consensus that it's not, suggestions on alternatives, and an ambition for
> faster and more stable; but not a definition of the thing is the reason why
> we must change?    Not saying we shouldn't, but I'm just asking so we try
> not to repeat the same mistakes following new steps.  Usually I've found
> that release blockage is derived from not defining clear objectives and
> definitions of good enough (not necessarily perfect) consistently
> throughout a release cycle.  It's not enough to start a release cycle with
> one scope of both, yet at the end continue to revise that definition.
>
> Jonathan G
>
>
> On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:
>
>     Sorry, I have taken too long to respond.
>
>     +1000000 on better CI and Testing, I think we all agree that needs to
> get
>     better and is a different beast than this topic.
>
>     As for the release branches, I am trying to follow a process similar to
>     ATS, which is usually the model we try to follow with our project.
>     While I agree that branching from master is the ideal solution, we
> tried
>     that and it didn't work.  I feel like we are trying to agree to do the
> same
>     thing that we already tried and didn't work.  I think we should have a
>     little freedom to commit things into master and, if they break things,
> work
>     on getting them fixed.  We also might commit PRs to master that are
> partial
>     implementations of a bigger feature.   If we are using a time based
>     strategy to cut our releases from master then we are going to be
>     continually worrying about making sure Master stays not broken.  I
> think we
>     should try to keep master deployable, but it's fine if it's not for
> short
>     periods of time.
>
>     I am not trying to suggest that we are "picky" about what goes into our
>     release branch.  I think we should try to cherry-pick as much as we can
>     into our release branch once it is ready to be cherry-picked.  I think
> we
>     should be able to take the tip of our release branch and deploy it
> whenever
>     we want with some confidence since we know that features/bug-fixes that
>     have been cherry-picked have already been reviewed and tested against
>     master.  Yes, its more work, but it also give us the ability to have
> more
>     confidence in what is in our release at any given time.  Again, we
> should
>     be trying to cherry-pick in everything that makes sense to cherry-pick.
>
>     I think the process that I would like to see looks like this:
>     - Cut a 4.x branch from Master
>     - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> branch
>     - Tag the 4.0.0 release from the 4.0.x branch
>     - New non-breaking changes and bug fixes that go into master are also
>     Cherry Picked to 4.x
>     - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when
> we
>     are ready for a RC.
>     - Rinse and repeat for all 4.x releases
>     - Once there's a need for a major release we make a 5.x branch from
> master
>
>     If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> then
>     4.0.x which we use to create the new 4.0.1 release (assuming we don't
> have
>     4.1 yet).  Yes, that is a more work than today, but I think A) it's not
>     really that bad and B) it gives us more control.
>
>     This should allow us to deploy much quicker and with more confidence
> than
>     is possible today.   We can also choose to deploy from any point in
> time
>     from the 4.x branch with relatively high confidence.
>
>     I'll wrap this up by saying that I don't think there's a perfect
> solution,
>     but I think we can agree on something that is worth trying at least
> with
>     the 4.x release life cycle.  Once we have some experience, we can
> figure
>     out where it is not meeting our needs and figure out how to adjust as
>     necessary.  I think we can all agree our current process is not
> working.
>
>     Thanks,
>     Dave
>
>
>
>     On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> wrote:
>
>     > +1 on more frequent releases
>     > +1 on improving our CI. We need to get our CI to the point where
> we're
>     > confident deploying changes to production if the CI passes.
>     > +1 on "trunk-based" development as opposed to GitFlow
>     > -1 on cherry-picking new features. This is risky, tedious, and
>     > represents unnecessary effort that would be better spent elsewhere,
>     > like improving our CI or adding more automated tests.
>     > +1 on cutting minor release branches directly off master
>     >
>     > I think cutting minor release branches directly off master can put us
>     > in a win-win scenario. Our releases become smaller and more frequent,
>     > and we avoid the unnecessary risk and effort of cherry-picking new
>     > features onto stale release branches.
>     >
>     > This is the process that I would propose:
>     > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
>     > 2. Develop master for 4-6 weeks.
>     > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
>     > 4. Repeat steps 2 and 3 for new minor releases as necessary.
>     > 5. Whenever a major (show-stopping) bug or security issue is found,
>     > cherry-pick the fix from master into the latest release branch (e.g.
>     > 4.1.x), and create a 4.1.1 RC.
>     >
>     > As the 4.x release manager, I really don't want to get into the
>     > business of selecting what should and should not go into a particular
>     > release. Every 4-6 weeks, we should just cut a new minor release
>     > branch off master and take everything that was contributed during
> that
>     > time, as long as everything builds fine and all the tests pass. The
>     > master branch, while it doesn't always have to be perfect, should at
>     > least always be deployable. If a new feature is merged with bugs and
>     > becomes a show-stopper, it should be either immediately remedied or
>     > reverted on master while we work on a fix.
>     >
>     > - Rawlin
>     >
>     > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
>     > <Mi...@comcast.com> wrote:
>     > >
>     > > In my opinion, in order to get to a cadence we are discussing we
> need to
>     > put a lot more work into the CI system. It has been failing
> consistently,
>     > doesn’t block PR approvals/merges and when it actually runs it does
> not
>     > test anything besides build and license headers.
>     > >
>     > > In the past couple of weeks we have had (off the top of my head):
>     > >
>     > > * Unit tests broken
>     > > * TO API tests broken
>     > > * Golang vendor issues
>     > > * TO Go build issues
>     > >
>     > > All were merged and broke master.
>     > >
>     > > Yes, having release branches sounds great and more aggressive
> cadence
>     > will minimize the amount of risk but I think that comes with the
> need to
>     > improve on our automated validation and testing.
>     > >
>     > > We have proven that we have not kept master in a good state and
> adding
>     > more releases (more branches) will make it even harder to keep that
>     > stability we are already fighting.
>     > >
>     > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> wrote:
>     > >
>     > >     Yeah, I get it. No one company should be driving release
>     > schedules/scope.
>     > >
>     > >     What I was really getting at is if ANY company has a recent
> version
>     > in a
>     > >     test or prod-like enviro (meaning the version is being
> exercised and
>     > >     thoroughly tested), then maybe we consider simply cutting a
> release
>     > from
>     > >     that version. For example, imagine company X had this version
> in a
>     > >     test/prod enviro:
>     > >
>     > >     Master-10287.7e62d07 (
>     > >
>     >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
>     > )
>     > >
>     > >     Then I would be in favor of cutting the 4.0 release from
>     > >     Master-10287.7e62d07 as we know it is proven to work based on
>     > company X's
>     > >     actual use of it.
>     > >
>     > >     The truth is (imo), we don't have the bandwidth to manually
> verify
>     > releases
>     > >     that we are not using/have not used nor do we have the
> necessary
>     > automated
>     > >     test coverage to verify these releases. So most of our
> releases are
>     > >     significantly untested.
>     > >
>     > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> ocket8888@gmail.com>
>     > wrote:
>     > >
>     > >     > I'm really not a fan of allowing Comcast to dictate the
> release
>     > scope and
>     > >     > schedule. If cherry-picking is too messy, then we can just
> cut new
>     > minor
>     > >     > releases directly from master.
>     > >     >
>     > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> mitchell852@gmail.com>
>     > wrote:
>     > >     >
>     > >     > > I don't think it's as easy as cherry picking (backporting)
>     > certain
>     > >     > features
>     > >     > > into a release branch. I could be wrong but I really don't
> think
>     > it is.
>     > >     > >
>     > >     > > So what I'm hearing is that 4.0 gets cut from master and
> we go
>     > through
>     > >     > with
>     > >     > > our normal process of testing, validating, etc. At the same
>     > time, 4.1
>     > >     > > branch is created from 4.0. Master moves on as normal.
> Let's say
>     > 25
>     > >     > commits
>     > >     > > come in to master during the next 6 weeks. During that 6
> weeks,
>     > we cherry
>     > >     > > pick only the features that look interesting for 4.1? (the
> rest
>     > just stay
>     > >     > > in master and will get picked up in 5.0?)
>     > >     > >
>     > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
>     > weeks apart
>     > >     > and
>     > >     > > have only "interesting" features so the releases are
>     > incrementally very
>     > >     > > small and much easier to test/validate, however, those
> features
>     > might
>     > >     > > depend on other commits. At some point, cherry picking
> becomes
>     > impossible
>     > >     > > as the foundation of the code base has shifted so
> drastically.
>     > Cherry
>     > >     > > picking will become very messy and introduce a high level
> of
>     > risk imo.
>     > >     > >
>     > >     > > Also, you'd have to manage the scope of these minor
> releases
>     > which we are
>     > >     > > not very good at because the designated release manager
> has a
>     > full-time
>     > >     > job
>     > >     > > to attend to and now we have even more releases to
>     > >     > > scope/test/validate/approve. I like the idea of faster
> releases
>     > but this
>     > >     > > sounds like more work that nobody has the bandwidth for.
>     > >     > >
>     > >     > > Ok, so rather than shooting down an idea with no
> alternative
>     > proposal, I
>     > >     > > propose this. At Comcast, we are trying to get to the point
>     > where we
>     > >     > > release TC to prod from the head of master every few
> weeks. What
>     > if every
>     > >     > > time Comcast does an internal upgrade, they note the
> commit hash
>     > and once
>     > >     > > comcast is happy with the upgrade in their prod enviro,
> they
>     > propose a
>     > >     > > release pinned to that commit hash? With this approach
> you'd end
>     > up with:
>     > >     > >
>     > >     > > 1. fast releases (hopefully every few weeks)
>     > >     > > 2. well tested releases as it's in a real prod enviro
>     > >     > > 3. little/no validation work for anyone
>     > >     > >
>     > >     > > Also, this doesn't have to be Comcast. If anyone else is
> out
>     > front at the
>     > >     > > edge of master and running it in a trusted enviro, they
> could
>     > propose a
>     > >     > > release as well.
>     > >     > >
>     > >     > > Jeremy
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
>     > david.neuman64@gmail.com>
>     > >     > > wrote:
>     > >     > >
>     > >     > > > I have been thinking about how we can get better with our
>     > release
>     > >     > > cadence.
>     > >     > > > I feel like we have slowed to a crawl and not been as
> good as
>     > we should
>     > >     > > > about how we release.  Our last Major release was in
> March and
>     > we
>     > >     > haven't
>     > >     > > > had a real release since.   Moving forward I would like
> to see
>     > us get
>     > >     > on
>     > >     > > a
>     > >     > > > more consistent cadence and provide smaller releases
> that can
>     > be more
>     > >     > > > easily digested by the community.  It's my hope that we
> can
>     > get to a
>     > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> we are
>     > >     > releasing
>     > >     > > > in such a way that not all releases all required (e.g.
> if you
>     > are on
>     > >     > 4.0
>     > >     > > > you don't need 4.1, you can just install 4.2), and we
> are more
>     > >     > deliberate
>     > >     > > > about what we put into a release.
>     > >     > > >
>     > >     > > > I think we can accomplish this by using our release
> branches
>     > better.
>     > >     > For
>     > >     > > > each major release we will cut a release branch from
> master
>     > (e.g. 4.x)
>     > >     > > and
>     > >     > > > then we will use cherry-picking to add new features and
> bug
>     > fixes to
>     > >     > that
>     > >     > > > release.  This means that if 4.0 has been released and
> you
>     > want to get
>     > >     > > your
>     > >     > > > feature/bug fix in 4.1, you will first submit your PR to
>     > master and
>     > >     > then
>     > >     > > > cheery pick your squash merged commit to the 4.x branch
> which
>     > we will
>     > >     > use
>     > >     > > > to create the 4.1 release.  I think we should allow
> either the
>     > >     > > contributor
>     > >     > > > or the committer (who is merging the PR) to suggest if a
>     > feature goes
>     > >     > > into
>     > >     > > > the release branch, and if we disagree we can take it to
> the
>     > list.  If
>     > >     > we
>     > >     > > > decide that every PR to master also goes into 4.1 then
> so be
>     > it, but at
>     > >     > > > least we are making a conscious decision of what goes
> into the
>     > next
>     > >     > > > release.   This will allow us to not be so worried about
> what
>     > we are
>     > >     > > > merging into master and how that will affect our next
> release.
>     > >     > According
>     > >     > > > to our 6 week cadence we will cut a new release from the
>     > current
>     > >     > feature
>     > >     > > > branch and put that up for a vote.  These releases will
> be
>     > small and
>     > >     > > > testable enough that we can get feedback quickly and
> move from
>     > RC to
>     > >     > > > release in a short time.  If a release has too many
> issues
>     > then we may
>     > >     > > end
>     > >     > > > up deciding to skip that minor release in favor of the
> next
>     > one,
>     > >     > > hopefully
>     > >     > > > this does not happen very often if at all.
>     > >     > > >
>     > >     > > > Once a breaking change is introduced to master which will
>     > require a new
>     > >     > > > major release, we will create a new major release branch
> from
>     > master
>     > >     > > (which
>     > >     > > > will mean a new release manager).  We will then repeat
> the
>     > same process
>     > >     > > > with the new release branch, etc, etc.
>     > >     > > >
>     > >     > > > As for LTS we will provide support for the latest major
>     > release plus
>     > >     > the
>     > >     > > > one previous.  So, once we release 4.0 we will support
> 4.0 and
>     > 3.1.
>     > >     > Any
>     > >     > > > security issues that arise will -- if present -- be
> applied to
>     > each of
>     > >     > > > these versions.  Once 4.1 is released we will support
> 4.1 and
>     > 3.1, etc,
>     > >     > > > etc.
>     > >     > > >
>     > >     > > > Please let me know if you have any thoughts on this
> plan.  If
>     > there are
>     > >     > > no
>     > >     > > > major objections I would like to try this with 4.x with
> the
>     > idea that
>     > >     > we
>     > >     > > > will adjust how we do things as necessary.
>     > >     > > >
>     > >     > > > We need to get better at releasing and something needs to
>     > change,
>     > >     > > hopefully
>     > >     > > > this can get us going in the right direction.
>     > >     > > >
>     > >     > > > Thanks,
>     > >     > > > Dave
>     > >     > > >
>     > >     > > >
>     > >     > > > TL;DR -  I am proposing a 6 week release cycle using
>     > cherry-picks to
>     > >     > our
>     > >     > > > release branch to control what goes into a minor release.
>     > Major
>     > >     > releases
>     > >     > > > will be cut from master as necessary - such as if there
> is a
>     > breaking
>     > >     > > > change that is introduced.
>     > >     > > >
>     > >     > >
>     > >     >
>     > >
>     > >
>     >
>
>
>

Re: [EXTERNAL] Re: Release Process

[Dave Neuman <ne...@apache.org>]

Thanks Brennan.  I agree that cherry picking can be scary, but it doesn’t
have to be. I hear you on the risks but I think we need to start
somewhere.  I feel like this won’t be as in practice as it may seem now and
we may be dealing with some FUD. I could also be absolutely wrong and
pushing a totally stupid idea :). We won’t know until we try.

Not to muddy the water but I think it’s worth noting that if we feel like
master has what we want when we are ready to cut a release, we can always
rebase it into the release branch. I think that’s reasonable and will
basically allow us to build our release from master.

On Wed, Oct 30, 2019 at 16:41 ocket 8888 <oc...@gmail.com> wrote:

> Cherry picking is scary. It will always have some associated risk. I
> understand that the point is to emulate pulling commits out rather than
> picking ones in, so that incomplete features and risky changes can be left
> out of a release that otherwise ought to be all-encompassing. However, that
> means essentially asking people to develop against the commit at which the
> last release branch was cut rather than master, because all changes need to
> "stand on their own," since without being a part of the review process for
> every PR that comes in no single person can be totally sure when they start
> work how much of the work between then and now can be considered "stable"
> enough to make the cut - even if that is a low bar. Which isn't inherently
> bad, it's just that it can lead to duplication of effort, and it's just not
> something we normally do as a community. Currently, nearly all development
> and testing outside of internal patching is taking place on master, so for
> what it's worth we already are somewhat concerned with it being stable all
> the time (please ignore failing integration tests on master at the time of
> this writing).
>
> So that's my opinion on cherry-picking. I do, though, think this represents
> a step in the right direction, even with cherry-picking. More frequent
> releases with better testing will lead to general improvements to the
> automated testing infrastructure, the ability to  kill off legacy code in
> our lifetimes, and make the stability of new releases more
> certain.Time-based release cycles get a big +1 from me, and if I have to
> live with cherry-picking to get that then so be it.
>
> On Wed, Oct 30, 2019 at 3:47 PM Dave Neuman <ne...@apache.org> wrote:
>
> > There are many things that are not working.  First of all we do not have
> > any semblance of a consistent release pattern.  If I get my feature
> merged
> > into Traffic Control today, I have no idea when that'll actually be in a
> > release.  Secondly, we do not currently have any way to limit what goes
> > into a release besides "everything in master since the last release".
> This
> > leads to a lot of uncertainty with new releases and lower confidence when
> > deploying a new release. This also forces us to not be able to move fast
> > when merging things into master since we are always trying to keep master
> > "stable".    Finally, since our releases are so big and infrequent it
> takes
> > a long time for the majority of the community to upgrade, which then
> > requires us to support old releases for a longer period of time.  Our
> > current available release is 3.0 which means there are still folks that
> are
> > on 2.2 and we should technically support it since it's n-1.  For
> reference,
> > 2.2 was released in June of 2018.
> > I agree that not having clear objectives and definitions of good enough
> > lead to problems.  I don't think it's realistic to plan all of the
> features
> > that we want to put into each release, it's an open source project and
> > that's not how it works.  This is why I am suggesting a consistent time
> > based release cycle.  This will allow us to all have the same
> understanding
> > of when a release is going to be and when our features will be included
> in
> > a release.  This also allows us to have smaller releases that are more
> > easily digestible which allows us to hopefully move faster as a
> community.
> >
> > BTW, 6 weeks was just a time period that felt right.  Just like with the
> > process if that doesn't work (too fast, too slow, etc) we can adjust that
> > as well.
> >
> > On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <
> Jonathan_Gray@comcast.com>
> > wrote:
> >
> > > Did we define why/how what we've been doing isnt' working?  I see a
> > > consensus that it's not, suggestions on alternatives, and an ambition
> for
> > > faster and more stable; but not a definition of the thing is the reason
> > why
> > > we must change?    Not saying we shouldn't, but I'm just asking so we
> try
> > > not to repeat the same mistakes following new steps.  Usually I've
> found
> > > that release blockage is derived from not defining clear objectives and
> > > definitions of good enough (not necessarily perfect) consistently
> > > throughout a release cycle.  It's not enough to start a release cycle
> > with
> > > one scope of both, yet at the end continue to revise that definition.
> > >
> > > Jonathan G
> > >
> > >
> > > On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:
> > >
> > >     Sorry, I have taken too long to respond.
> > >
> > >     +1000000 on better CI and Testing, I think we all agree that needs
> to
> > > get
> > >     better and is a different beast than this topic.
> > >
> > >     As for the release branches, I am trying to follow a process
> similar
> > to
> > >     ATS, which is usually the model we try to follow with our project.
> > >     While I agree that branching from master is the ideal solution, we
> > > tried
> > >     that and it didn't work.  I feel like we are trying to agree to do
> > the
> > > same
> > >     thing that we already tried and didn't work.  I think we should
> have
> > a
> > >     little freedom to commit things into master and, if they break
> > things,
> > > work
> > >     on getting them fixed.  We also might commit PRs to master that are
> > > partial
> > >     implementations of a bigger feature.   If we are using a time based
> > >     strategy to cut our releases from master then we are going to be
> > >     continually worrying about making sure Master stays not broken.  I
> > > think we
> > >     should try to keep master deployable, but it's fine if it's not for
> > > short
> > >     periods of time.
> > >
> > >     I am not trying to suggest that we are "picky" about what goes into
> > our
> > >     release branch.  I think we should try to cherry-pick as much as we
> > can
> > >     into our release branch once it is ready to be cherry-picked.  I
> > think
> > > we
> > >     should be able to take the tip of our release branch and deploy it
> > > whenever
> > >     we want with some confidence since we know that features/bug-fixes
> > that
> > >     have been cherry-picked have already been reviewed and tested
> against
> > >     master.  Yes, its more work, but it also give us the ability to
> have
> > > more
> > >     confidence in what is in our release at any given time.  Again, we
> > > should
> > >     be trying to cherry-pick in everything that makes sense to
> > cherry-pick.
> > >
> > >     I think the process that I would like to see looks like this:
> > >     - Cut a 4.x branch from Master
> > >     - Once we think we are ready for a RC cut a 4.0.x branch from the
> 4.x
> > > branch
> > >     - Tag the 4.0.0 release from the 4.0.x branch
> > >     - New non-breaking changes and bug fixes that go into master are
> also
> > >     Cherry Picked to 4.x
> > >     - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag
> when
> > > we
> > >     are ready for a RC.
> > >     - Rinse and repeat for all 4.x releases
> > >     - Once there's a need for a major release we make a 5.x branch from
> > > master
> > >
> > >     If we find a bug in 4.0.0 then we commit the fix to master, then
> 4.x,
> > > then
> > >     4.0.x which we use to create the new 4.0.1 release (assuming we
> don't
> > > have
> > >     4.1 yet).  Yes, that is a more work than today, but I think A) it's
> > not
> > >     really that bad and B) it gives us more control.
> > >
> > >     This should allow us to deploy much quicker and with more
> confidence
> > > than
> > >     is possible today.   We can also choose to deploy from any point in
> > > time
> > >     from the 4.x branch with relatively high confidence.
> > >
> > >     I'll wrap this up by saying that I don't think there's a perfect
> > > solution,
> > >     but I think we can agree on something that is worth trying at least
> > > with
> > >     the 4.x release life cycle.  Once we have some experience, we can
> > > figure
> > >     out where it is not meeting our needs and figure out how to adjust
> as
> > >     necessary.  I think we can all agree our current process is not
> > > working.
> > >
> > >     Thanks,
> > >     Dave
> > >
> > >
> > >
> > >     On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > > wrote:
> > >
> > >     > +1 on more frequent releases
> > >     > +1 on improving our CI. We need to get our CI to the point where
> > > we're
> > >     > confident deploying changes to production if the CI passes.
> > >     > +1 on "trunk-based" development as opposed to GitFlow
> > >     > -1 on cherry-picking new features. This is risky, tedious, and
> > >     > represents unnecessary effort that would be better spent
> elsewhere,
> > >     > like improving our CI or adding more automated tests.
> > >     > +1 on cutting minor release branches directly off master
> > >     >
> > >     > I think cutting minor release branches directly off master can
> put
> > us
> > >     > in a win-win scenario. Our releases become smaller and more
> > frequent,
> > >     > and we avoid the unnecessary risk and effort of cherry-picking
> new
> > >     > features onto stale release branches.
> > >     >
> > >     > This is the process that I would propose:
> > >     > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > >     > 2. Develop master for 4-6 weeks.
> > >     > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > >     > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > >     > 5. Whenever a major (show-stopping) bug or security issue is
> found,
> > >     > cherry-pick the fix from master into the latest release branch
> > (e.g.
> > >     > 4.1.x), and create a 4.1.1 RC.
> > >     >
> > >     > As the 4.x release manager, I really don't want to get into the
> > >     > business of selecting what should and should not go into a
> > particular
> > >     > release. Every 4-6 weeks, we should just cut a new minor release
> > >     > branch off master and take everything that was contributed during
> > > that
> > >     > time, as long as everything builds fine and all the tests pass.
> The
> > >     > master branch, while it doesn't always have to be perfect, should
> > at
> > >     > least always be deployable. If a new feature is merged with bugs
> > and
> > >     > becomes a show-stopper, it should be either immediately remedied
> or
> > >     > reverted on master while we work on a fix.
> > >     >
> > >     > - Rawlin
> > >     >
> > >     > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > >     > <Mi...@comcast.com> wrote:
> > >     > >
> > >     > > In my opinion, in order to get to a cadence we are discussing
> we
> > > need to
> > >     > put a lot more work into the CI system. It has been failing
> > > consistently,
> > >     > doesn’t block PR approvals/merges and when it actually runs it
> does
> > > not
> > >     > test anything besides build and license headers.
> > >     > >
> > >     > > In the past couple of weeks we have had (off the top of my
> head):
> > >     > >
> > >     > > * Unit tests broken
> > >     > > * TO API tests broken
> > >     > > * Golang vendor issues
> > >     > > * TO Go build issues
> > >     > >
> > >     > > All were merged and broke master.
> > >     > >
> > >     > > Yes, having release branches sounds great and more aggressive
> > > cadence
> > >     > will minimize the amount of risk but I think that comes with the
> > > need to
> > >     > improve on our automated validation and testing.
> > >     > >
> > >     > > We have proven that we have not kept master in a good state and
> > > adding
> > >     > more releases (more branches) will make it even harder to keep
> that
> > >     > stability we are already fighting.
> > >     > >
> > >     > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mitchell852@gmail.com
> >
> > > wrote:
> > >     > >
> > >     > >     Yeah, I get it. No one company should be driving release
> > >     > schedules/scope.
> > >     > >
> > >     > >     What I was really getting at is if ANY company has a recent
> > > version
> > >     > in a
> > >     > >     test or prod-like enviro (meaning the version is being
> > > exercised and
> > >     > >     thoroughly tested), then maybe we consider simply cutting a
> > > release
> > >     > from
> > >     > >     that version. For example, imagine company X had this
> version
> > > in a
> > >     > >     test/prod enviro:
> > >     > >
> > >     > >     Master-10287.7e62d07 (
> > >     > >
> > >     >
> > >
> >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > >     > )
> > >     > >
> > >     > >     Then I would be in favor of cutting the 4.0 release from
> > >     > >     Master-10287.7e62d07 as we know it is proven to work based
> on
> > >     > company X's
> > >     > >     actual use of it.
> > >     > >
> > >     > >     The truth is (imo), we don't have the bandwidth to manually
> > > verify
> > >     > releases
> > >     > >     that we are not using/have not used nor do we have the
> > > necessary
> > >     > automated
> > >     > >     test coverage to verify these releases. So most of our
> > > releases are
> > >     > >     significantly untested.
> > >     > >
> > >     > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> > > ocket8888@gmail.com>
> > >     > wrote:
> > >     > >
> > >     > >     > I'm really not a fan of allowing Comcast to dictate the
> > > release
> > >     > scope and
> > >     > >     > schedule. If cherry-picking is too messy, then we can
> just
> > > cut new
> > >     > minor
> > >     > >     > releases directly from master.
> > >     > >     >
> > >     > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > > mitchell852@gmail.com>
> > >     > wrote:
> > >     > >     >
> > >     > >     > > I don't think it's as easy as cherry picking
> > (backporting)
> > >     > certain
> > >     > >     > features
> > >     > >     > > into a release branch. I could be wrong but I really
> > don't
> > > think
> > >     > it is.
> > >     > >     > >
> > >     > >     > > So what I'm hearing is that 4.0 gets cut from master
> and
> > > we go
> > >     > through
> > >     > >     > with
> > >     > >     > > our normal process of testing, validating, etc. At the
> > same
> > >     > time, 4.1
> > >     > >     > > branch is created from 4.0. Master moves on as normal.
> > > Let's say
> > >     > 25
> > >     > >     > commits
> > >     > >     > > come in to master during the next 6 weeks. During that
> 6
> > > weeks,
> > >     > we cherry
> > >     > >     > > pick only the features that look interesting for 4.1?
> > (the
> > > rest
> > >     > just stay
> > >     > >     > > in master and will get picked up in 5.0?)
> > >     > >     > >
> > >     > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> > roughly 6
> > >     > weeks apart
> > >     > >     > and
> > >     > >     > > have only "interesting" features so the releases are
> > >     > incrementally very
> > >     > >     > > small and much easier to test/validate, however, those
> > > features
> > >     > might
> > >     > >     > > depend on other commits. At some point, cherry picking
> > > becomes
> > >     > impossible
> > >     > >     > > as the foundation of the code base has shifted so
> > > drastically.
> > >     > Cherry
> > >     > >     > > picking will become very messy and introduce a high
> level
> > > of
> > >     > risk imo.
> > >     > >     > >
> > >     > >     > > Also, you'd have to manage the scope of these minor
> > > releases
> > >     > which we are
> > >     > >     > > not very good at because the designated release manager
> > > has a
> > >     > full-time
> > >     > >     > job
> > >     > >     > > to attend to and now we have even more releases to
> > >     > >     > > scope/test/validate/approve. I like the idea of faster
> > > releases
> > >     > but this
> > >     > >     > > sounds like more work that nobody has the bandwidth
> for.
> > >     > >     > >
> > >     > >     > > Ok, so rather than shooting down an idea with no
> > > alternative
> > >     > proposal, I
> > >     > >     > > propose this. At Comcast, we are trying to get to the
> > point
> > >     > where we
> > >     > >     > > release TC to prod from the head of master every few
> > > weeks. What
> > >     > if every
> > >     > >     > > time Comcast does an internal upgrade, they note the
> > > commit hash
> > >     > and once
> > >     > >     > > comcast is happy with the upgrade in their prod enviro,
> > > they
> > >     > propose a
> > >     > >     > > release pinned to that commit hash? With this approach
> > > you'd end
> > >     > up with:
> > >     > >     > >
> > >     > >     > > 1. fast releases (hopefully every few weeks)
> > >     > >     > > 2. well tested releases as it's in a real prod enviro
> > >     > >     > > 3. little/no validation work for anyone
> > >     > >     > >
> > >     > >     > > Also, this doesn't have to be Comcast. If anyone else
> is
> > > out
> > >     > front at the
> > >     > >     > > edge of master and running it in a trusted enviro, they
> > > could
> > >     > propose a
> > >     > >     > > release as well.
> > >     > >     > >
> > >     > >     > > Jeremy
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > >
> > >     > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > >     > david.neuman64@gmail.com>
> > >     > >     > > wrote:
> > >     > >     > >
> > >     > >     > > > I have been thinking about how we can get better with
> > our
> > >     > release
> > >     > >     > > cadence.
> > >     > >     > > > I feel like we have slowed to a crawl and not been as
> > > good as
> > >     > we should
> > >     > >     > > > about how we release.  Our last Major release was in
> > > March and
> > >     > we
> > >     > >     > haven't
> > >     > >     > > > had a real release since.   Moving forward I would
> like
> > > to see
> > >     > us get
> > >     > >     > on
> > >     > >     > > a
> > >     > >     > > > more consistent cadence and provide smaller releases
> > > that can
> > >     > be more
> > >     > >     > > > easily digested by the community.  It's my hope that
> we
> > > can
> > >     > get to a
> > >     > >     > > > cadence where we are doing releases every 4 to 6
> weeks,
> > > we are
> > >     > >     > releasing
> > >     > >     > > > in such a way that not all releases all required
> (e.g.
> > > if you
> > >     > are on
> > >     > >     > 4.0
> > >     > >     > > > you don't need 4.1, you can just install 4.2), and we
> > > are more
> > >     > >     > deliberate
> > >     > >     > > > about what we put into a release.
> > >     > >     > > >
> > >     > >     > > > I think we can accomplish this by using our release
> > > branches
> > >     > better.
> > >     > >     > For
> > >     > >     > > > each major release we will cut a release branch from
> > > master
> > >     > (e.g. 4.x)
> > >     > >     > > and
> > >     > >     > > > then we will use cherry-picking to add new features
> and
> > > bug
> > >     > fixes to
> > >     > >     > that
> > >     > >     > > > release.  This means that if 4.0 has been released
> and
> > > you
> > >     > want to get
> > >     > >     > > your
> > >     > >     > > > feature/bug fix in 4.1, you will first submit your PR
> > to
> > >     > master and
> > >     > >     > then
> > >     > >     > > > cheery pick your squash merged commit to the 4.x
> branch
> > > which
> > >     > we will
> > >     > >     > use
> > >     > >     > > > to create the 4.1 release.  I think we should allow
> > > either the
> > >     > >     > > contributor
> > >     > >     > > > or the committer (who is merging the PR) to suggest
> if
> > a
> > >     > feature goes
> > >     > >     > > into
> > >     > >     > > > the release branch, and if we disagree we can take it
> > to
> > > the
> > >     > list.  If
> > >     > >     > we
> > >     > >     > > > decide that every PR to master also goes into 4.1
> then
> > > so be
> > >     > it, but at
> > >     > >     > > > least we are making a conscious decision of what goes
> > > into the
> > >     > next
> > >     > >     > > > release.   This will allow us to not be so worried
> > about
> > > what
> > >     > we are
> > >     > >     > > > merging into master and how that will affect our next
> > > release.
> > >     > >     > According
> > >     > >     > > > to our 6 week cadence we will cut a new release from
> > the
> > >     > current
> > >     > >     > feature
> > >     > >     > > > branch and put that up for a vote.  These releases
> will
> > > be
> > >     > small and
> > >     > >     > > > testable enough that we can get feedback quickly and
> > > move from
> > >     > RC to
> > >     > >     > > > release in a short time.  If a release has too many
> > > issues
> > >     > then we may
> > >     > >     > > end
> > >     > >     > > > up deciding to skip that minor release in favor of
> the
> > > next
> > >     > one,
> > >     > >     > > hopefully
> > >     > >     > > > this does not happen very often if at all.
> > >     > >     > > >
> > >     > >     > > > Once a breaking change is introduced to master which
> > will
> > >     > require a new
> > >     > >     > > > major release, we will create a new major release
> > branch
> > > from
> > >     > master
> > >     > >     > > (which
> > >     > >     > > > will mean a new release manager).  We will then
> repeat
> > > the
> > >     > same process
> > >     > >     > > > with the new release branch, etc, etc.
> > >     > >     > > >
> > >     > >     > > > As for LTS we will provide support for the latest
> major
> > >     > release plus
> > >     > >     > the
> > >     > >     > > > one previous.  So, once we release 4.0 we will
> support
> > > 4.0 and
> > >     > 3.1.
> > >     > >     > Any
> > >     > >     > > > security issues that arise will -- if present -- be
> > > applied to
> > >     > each of
> > >     > >     > > > these versions.  Once 4.1 is released we will support
> > > 4.1 and
> > >     > 3.1, etc,
> > >     > >     > > > etc.
> > >     > >     > > >
> > >     > >     > > > Please let me know if you have any thoughts on this
> > > plan.  If
> > >     > there are
> > >     > >     > > no
> > >     > >     > > > major objections I would like to try this with 4.x
> with
> > > the
> > >     > idea that
> > >     > >     > we
> > >     > >     > > > will adjust how we do things as necessary.
> > >     > >     > > >
> > >     > >     > > > We need to get better at releasing and something
> needs
> > to
> > >     > change,
> > >     > >     > > hopefully
> > >     > >     > > > this can get us going in the right direction.
> > >     > >     > > >
> > >     > >     > > > Thanks,
> > >     > >     > > > Dave
> > >     > >     > > >
> > >     > >     > > >
> > >     > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > >     > cherry-picks to
> > >     > >     > our
> > >     > >     > > > release branch to control what goes into a minor
> > release.
> > >     > Major
> > >     > >     > releases
> > >     > >     > > > will be cut from master as necessary - such as if
> there
> > > is a
> > >     > breaking
> > >     > >     > > > change that is introduced.
> > >     > >     > > >
> > >     > >     > >
> > >     > >     >
> > >     > >
> > >     > >
> > >     >
> > >
> > >
> > >
> >
>

Re: [EXTERNAL] Re: Release Process

[ocket 8888 <oc...@gmail.com>]

Cherry picking is scary. It will always have some associated risk. I
understand that the point is to emulate pulling commits out rather than
picking ones in, so that incomplete features and risky changes can be left
out of a release that otherwise ought to be all-encompassing. However, that
means essentially asking people to develop against the commit at which the
last release branch was cut rather than master, because all changes need to
"stand on their own," since without being a part of the review process for
every PR that comes in no single person can be totally sure when they start
work how much of the work between then and now can be considered "stable"
enough to make the cut - even if that is a low bar. Which isn't inherently
bad, it's just that it can lead to duplication of effort, and it's just not
something we normally do as a community. Currently, nearly all development
and testing outside of internal patching is taking place on master, so for
what it's worth we already are somewhat concerned with it being stable all
the time (please ignore failing integration tests on master at the time of
this writing).

So that's my opinion on cherry-picking. I do, though, think this represents
a step in the right direction, even with cherry-picking. More frequent
releases with better testing will lead to general improvements to the
automated testing infrastructure, the ability to  kill off legacy code in
our lifetimes, and make the stability of new releases more
certain.Time-based release cycles get a big +1 from me, and if I have to
live with cherry-picking to get that then so be it.

On Wed, Oct 30, 2019 at 3:47 PM Dave Neuman <ne...@apache.org> wrote:

> There are many things that are not working.  First of all we do not have
> any semblance of a consistent release pattern.  If I get my feature merged
> into Traffic Control today, I have no idea when that'll actually be in a
> release.  Secondly, we do not currently have any way to limit what goes
> into a release besides "everything in master since the last release".  This
> leads to a lot of uncertainty with new releases and lower confidence when
> deploying a new release. This also forces us to not be able to move fast
> when merging things into master since we are always trying to keep master
> "stable".    Finally, since our releases are so big and infrequent it takes
> a long time for the majority of the community to upgrade, which then
> requires us to support old releases for a longer period of time.  Our
> current available release is 3.0 which means there are still folks that are
> on 2.2 and we should technically support it since it's n-1.  For reference,
> 2.2 was released in June of 2018.
> I agree that not having clear objectives and definitions of good enough
> lead to problems.  I don't think it's realistic to plan all of the features
> that we want to put into each release, it's an open source project and
> that's not how it works.  This is why I am suggesting a consistent time
> based release cycle.  This will allow us to all have the same understanding
> of when a release is going to be and when our features will be included in
> a release.  This also allows us to have smaller releases that are more
> easily digestible which allows us to hopefully move faster as a community.
>
> BTW, 6 weeks was just a time period that felt right.  Just like with the
> process if that doesn't work (too fast, too slow, etc) we can adjust that
> as well.
>
> On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <Jo...@comcast.com>
> wrote:
>
> > Did we define why/how what we've been doing isnt' working?  I see a
> > consensus that it's not, suggestions on alternatives, and an ambition for
> > faster and more stable; but not a definition of the thing is the reason
> why
> > we must change?    Not saying we shouldn't, but I'm just asking so we try
> > not to repeat the same mistakes following new steps.  Usually I've found
> > that release blockage is derived from not defining clear objectives and
> > definitions of good enough (not necessarily perfect) consistently
> > throughout a release cycle.  It's not enough to start a release cycle
> with
> > one scope of both, yet at the end continue to revise that definition.
> >
> > Jonathan G
> >
> >
> > On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:
> >
> >     Sorry, I have taken too long to respond.
> >
> >     +1000000 on better CI and Testing, I think we all agree that needs to
> > get
> >     better and is a different beast than this topic.
> >
> >     As for the release branches, I am trying to follow a process similar
> to
> >     ATS, which is usually the model we try to follow with our project.
> >     While I agree that branching from master is the ideal solution, we
> > tried
> >     that and it didn't work.  I feel like we are trying to agree to do
> the
> > same
> >     thing that we already tried and didn't work.  I think we should have
> a
> >     little freedom to commit things into master and, if they break
> things,
> > work
> >     on getting them fixed.  We also might commit PRs to master that are
> > partial
> >     implementations of a bigger feature.   If we are using a time based
> >     strategy to cut our releases from master then we are going to be
> >     continually worrying about making sure Master stays not broken.  I
> > think we
> >     should try to keep master deployable, but it's fine if it's not for
> > short
> >     periods of time.
> >
> >     I am not trying to suggest that we are "picky" about what goes into
> our
> >     release branch.  I think we should try to cherry-pick as much as we
> can
> >     into our release branch once it is ready to be cherry-picked.  I
> think
> > we
> >     should be able to take the tip of our release branch and deploy it
> > whenever
> >     we want with some confidence since we know that features/bug-fixes
> that
> >     have been cherry-picked have already been reviewed and tested against
> >     master.  Yes, its more work, but it also give us the ability to have
> > more
> >     confidence in what is in our release at any given time.  Again, we
> > should
> >     be trying to cherry-pick in everything that makes sense to
> cherry-pick.
> >
> >     I think the process that I would like to see looks like this:
> >     - Cut a 4.x branch from Master
> >     - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > branch
> >     - Tag the 4.0.0 release from the 4.0.x branch
> >     - New non-breaking changes and bug fixes that go into master are also
> >     Cherry Picked to 4.x
> >     - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when
> > we
> >     are ready for a RC.
> >     - Rinse and repeat for all 4.x releases
> >     - Once there's a need for a major release we make a 5.x branch from
> > master
> >
> >     If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > then
> >     4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > have
> >     4.1 yet).  Yes, that is a more work than today, but I think A) it's
> not
> >     really that bad and B) it gives us more control.
> >
> >     This should allow us to deploy much quicker and with more confidence
> > than
> >     is possible today.   We can also choose to deploy from any point in
> > time
> >     from the 4.x branch with relatively high confidence.
> >
> >     I'll wrap this up by saying that I don't think there's a perfect
> > solution,
> >     but I think we can agree on something that is worth trying at least
> > with
> >     the 4.x release life cycle.  Once we have some experience, we can
> > figure
> >     out where it is not meeting our needs and figure out how to adjust as
> >     necessary.  I think we can all agree our current process is not
> > working.
> >
> >     Thanks,
> >     Dave
> >
> >
> >
> >     On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > wrote:
> >
> >     > +1 on more frequent releases
> >     > +1 on improving our CI. We need to get our CI to the point where
> > we're
> >     > confident deploying changes to production if the CI passes.
> >     > +1 on "trunk-based" development as opposed to GitFlow
> >     > -1 on cherry-picking new features. This is risky, tedious, and
> >     > represents unnecessary effort that would be better spent elsewhere,
> >     > like improving our CI or adding more automated tests.
> >     > +1 on cutting minor release branches directly off master
> >     >
> >     > I think cutting minor release branches directly off master can put
> us
> >     > in a win-win scenario. Our releases become smaller and more
> frequent,
> >     > and we avoid the unnecessary risk and effort of cherry-picking new
> >     > features onto stale release branches.
> >     >
> >     > This is the process that I would propose:
> >     > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> >     > 2. Develop master for 4-6 weeks.
> >     > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> >     > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> >     > 5. Whenever a major (show-stopping) bug or security issue is found,
> >     > cherry-pick the fix from master into the latest release branch
> (e.g.
> >     > 4.1.x), and create a 4.1.1 RC.
> >     >
> >     > As the 4.x release manager, I really don't want to get into the
> >     > business of selecting what should and should not go into a
> particular
> >     > release. Every 4-6 weeks, we should just cut a new minor release
> >     > branch off master and take everything that was contributed during
> > that
> >     > time, as long as everything builds fine and all the tests pass. The
> >     > master branch, while it doesn't always have to be perfect, should
> at
> >     > least always be deployable. If a new feature is merged with bugs
> and
> >     > becomes a show-stopper, it should be either immediately remedied or
> >     > reverted on master while we work on a fix.
> >     >
> >     > - Rawlin
> >     >
> >     > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> >     > <Mi...@comcast.com> wrote:
> >     > >
> >     > > In my opinion, in order to get to a cadence we are discussing we
> > need to
> >     > put a lot more work into the CI system. It has been failing
> > consistently,
> >     > doesn’t block PR approvals/merges and when it actually runs it does
> > not
> >     > test anything besides build and license headers.
> >     > >
> >     > > In the past couple of weeks we have had (off the top of my head):
> >     > >
> >     > > * Unit tests broken
> >     > > * TO API tests broken
> >     > > * Golang vendor issues
> >     > > * TO Go build issues
> >     > >
> >     > > All were merged and broke master.
> >     > >
> >     > > Yes, having release branches sounds great and more aggressive
> > cadence
> >     > will minimize the amount of risk but I think that comes with the
> > need to
> >     > improve on our automated validation and testing.
> >     > >
> >     > > We have proven that we have not kept master in a good state and
> > adding
> >     > more releases (more branches) will make it even harder to keep that
> >     > stability we are already fighting.
> >     > >
> >     > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> > wrote:
> >     > >
> >     > >     Yeah, I get it. No one company should be driving release
> >     > schedules/scope.
> >     > >
> >     > >     What I was really getting at is if ANY company has a recent
> > version
> >     > in a
> >     > >     test or prod-like enviro (meaning the version is being
> > exercised and
> >     > >     thoroughly tested), then maybe we consider simply cutting a
> > release
> >     > from
> >     > >     that version. For example, imagine company X had this version
> > in a
> >     > >     test/prod enviro:
> >     > >
> >     > >     Master-10287.7e62d07 (
> >     > >
> >     >
> >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> >     > )
> >     > >
> >     > >     Then I would be in favor of cutting the 4.0 release from
> >     > >     Master-10287.7e62d07 as we know it is proven to work based on
> >     > company X's
> >     > >     actual use of it.
> >     > >
> >     > >     The truth is (imo), we don't have the bandwidth to manually
> > verify
> >     > releases
> >     > >     that we are not using/have not used nor do we have the
> > necessary
> >     > automated
> >     > >     test coverage to verify these releases. So most of our
> > releases are
> >     > >     significantly untested.
> >     > >
> >     > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> > ocket8888@gmail.com>
> >     > wrote:
> >     > >
> >     > >     > I'm really not a fan of allowing Comcast to dictate the
> > release
> >     > scope and
> >     > >     > schedule. If cherry-picking is too messy, then we can just
> > cut new
> >     > minor
> >     > >     > releases directly from master.
> >     > >     >
> >     > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > mitchell852@gmail.com>
> >     > wrote:
> >     > >     >
> >     > >     > > I don't think it's as easy as cherry picking
> (backporting)
> >     > certain
> >     > >     > features
> >     > >     > > into a release branch. I could be wrong but I really
> don't
> > think
> >     > it is.
> >     > >     > >
> >     > >     > > So what I'm hearing is that 4.0 gets cut from master and
> > we go
> >     > through
> >     > >     > with
> >     > >     > > our normal process of testing, validating, etc. At the
> same
> >     > time, 4.1
> >     > >     > > branch is created from 4.0. Master moves on as normal.
> > Let's say
> >     > 25
> >     > >     > commits
> >     > >     > > come in to master during the next 6 weeks. During that 6
> > weeks,
> >     > we cherry
> >     > >     > > pick only the features that look interesting for 4.1?
> (the
> > rest
> >     > just stay
> >     > >     > > in master and will get picked up in 5.0?)
> >     > >     > >
> >     > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> roughly 6
> >     > weeks apart
> >     > >     > and
> >     > >     > > have only "interesting" features so the releases are
> >     > incrementally very
> >     > >     > > small and much easier to test/validate, however, those
> > features
> >     > might
> >     > >     > > depend on other commits. At some point, cherry picking
> > becomes
> >     > impossible
> >     > >     > > as the foundation of the code base has shifted so
> > drastically.
> >     > Cherry
> >     > >     > > picking will become very messy and introduce a high level
> > of
> >     > risk imo.
> >     > >     > >
> >     > >     > > Also, you'd have to manage the scope of these minor
> > releases
> >     > which we are
> >     > >     > > not very good at because the designated release manager
> > has a
> >     > full-time
> >     > >     > job
> >     > >     > > to attend to and now we have even more releases to
> >     > >     > > scope/test/validate/approve. I like the idea of faster
> > releases
> >     > but this
> >     > >     > > sounds like more work that nobody has the bandwidth for.
> >     > >     > >
> >     > >     > > Ok, so rather than shooting down an idea with no
> > alternative
> >     > proposal, I
> >     > >     > > propose this. At Comcast, we are trying to get to the
> point
> >     > where we
> >     > >     > > release TC to prod from the head of master every few
> > weeks. What
> >     > if every
> >     > >     > > time Comcast does an internal upgrade, they note the
> > commit hash
> >     > and once
> >     > >     > > comcast is happy with the upgrade in their prod enviro,
> > they
> >     > propose a
> >     > >     > > release pinned to that commit hash? With this approach
> > you'd end
> >     > up with:
> >     > >     > >
> >     > >     > > 1. fast releases (hopefully every few weeks)
> >     > >     > > 2. well tested releases as it's in a real prod enviro
> >     > >     > > 3. little/no validation work for anyone
> >     > >     > >
> >     > >     > > Also, this doesn't have to be Comcast. If anyone else is
> > out
> >     > front at the
> >     > >     > > edge of master and running it in a trusted enviro, they
> > could
> >     > propose a
> >     > >     > > release as well.
> >     > >     > >
> >     > >     > > Jeremy
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > >
> >     > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> >     > david.neuman64@gmail.com>
> >     > >     > > wrote:
> >     > >     > >
> >     > >     > > > I have been thinking about how we can get better with
> our
> >     > release
> >     > >     > > cadence.
> >     > >     > > > I feel like we have slowed to a crawl and not been as
> > good as
> >     > we should
> >     > >     > > > about how we release.  Our last Major release was in
> > March and
> >     > we
> >     > >     > haven't
> >     > >     > > > had a real release since.   Moving forward I would like
> > to see
> >     > us get
> >     > >     > on
> >     > >     > > a
> >     > >     > > > more consistent cadence and provide smaller releases
> > that can
> >     > be more
> >     > >     > > > easily digested by the community.  It's my hope that we
> > can
> >     > get to a
> >     > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> > we are
> >     > >     > releasing
> >     > >     > > > in such a way that not all releases all required (e.g.
> > if you
> >     > are on
> >     > >     > 4.0
> >     > >     > > > you don't need 4.1, you can just install 4.2), and we
> > are more
> >     > >     > deliberate
> >     > >     > > > about what we put into a release.
> >     > >     > > >
> >     > >     > > > I think we can accomplish this by using our release
> > branches
> >     > better.
> >     > >     > For
> >     > >     > > > each major release we will cut a release branch from
> > master
> >     > (e.g. 4.x)
> >     > >     > > and
> >     > >     > > > then we will use cherry-picking to add new features and
> > bug
> >     > fixes to
> >     > >     > that
> >     > >     > > > release.  This means that if 4.0 has been released and
> > you
> >     > want to get
> >     > >     > > your
> >     > >     > > > feature/bug fix in 4.1, you will first submit your PR
> to
> >     > master and
> >     > >     > then
> >     > >     > > > cheery pick your squash merged commit to the 4.x branch
> > which
> >     > we will
> >     > >     > use
> >     > >     > > > to create the 4.1 release.  I think we should allow
> > either the
> >     > >     > > contributor
> >     > >     > > > or the committer (who is merging the PR) to suggest if
> a
> >     > feature goes
> >     > >     > > into
> >     > >     > > > the release branch, and if we disagree we can take it
> to
> > the
> >     > list.  If
> >     > >     > we
> >     > >     > > > decide that every PR to master also goes into 4.1 then
> > so be
> >     > it, but at
> >     > >     > > > least we are making a conscious decision of what goes
> > into the
> >     > next
> >     > >     > > > release.   This will allow us to not be so worried
> about
> > what
> >     > we are
> >     > >     > > > merging into master and how that will affect our next
> > release.
> >     > >     > According
> >     > >     > > > to our 6 week cadence we will cut a new release from
> the
> >     > current
> >     > >     > feature
> >     > >     > > > branch and put that up for a vote.  These releases will
> > be
> >     > small and
> >     > >     > > > testable enough that we can get feedback quickly and
> > move from
> >     > RC to
> >     > >     > > > release in a short time.  If a release has too many
> > issues
> >     > then we may
> >     > >     > > end
> >     > >     > > > up deciding to skip that minor release in favor of the
> > next
> >     > one,
> >     > >     > > hopefully
> >     > >     > > > this does not happen very often if at all.
> >     > >     > > >
> >     > >     > > > Once a breaking change is introduced to master which
> will
> >     > require a new
> >     > >     > > > major release, we will create a new major release
> branch
> > from
> >     > master
> >     > >     > > (which
> >     > >     > > > will mean a new release manager).  We will then repeat
> > the
> >     > same process
> >     > >     > > > with the new release branch, etc, etc.
> >     > >     > > >
> >     > >     > > > As for LTS we will provide support for the latest major
> >     > release plus
> >     > >     > the
> >     > >     > > > one previous.  So, once we release 4.0 we will support
> > 4.0 and
> >     > 3.1.
> >     > >     > Any
> >     > >     > > > security issues that arise will -- if present -- be
> > applied to
> >     > each of
> >     > >     > > > these versions.  Once 4.1 is released we will support
> > 4.1 and
> >     > 3.1, etc,
> >     > >     > > > etc.
> >     > >     > > >
> >     > >     > > > Please let me know if you have any thoughts on this
> > plan.  If
> >     > there are
> >     > >     > > no
> >     > >     > > > major objections I would like to try this with 4.x with
> > the
> >     > idea that
> >     > >     > we
> >     > >     > > > will adjust how we do things as necessary.
> >     > >     > > >
> >     > >     > > > We need to get better at releasing and something needs
> to
> >     > change,
> >     > >     > > hopefully
> >     > >     > > > this can get us going in the right direction.
> >     > >     > > >
> >     > >     > > > Thanks,
> >     > >     > > > Dave
> >     > >     > > >
> >     > >     > > >
> >     > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> >     > cherry-picks to
> >     > >     > our
> >     > >     > > > release branch to control what goes into a minor
> release.
> >     > Major
> >     > >     > releases
> >     > >     > > > will be cut from master as necessary - such as if there
> > is a
> >     > breaking
> >     > >     > > > change that is introduced.
> >     > >     > > >
> >     > >     > >
> >     > >     >
> >     > >
> >     > >
> >     >
> >
> >
> >
>

Re: [EXTERNAL] Re: Release Process

[Dave Neuman <ne...@apache.org>]

There are many things that are not working.  First of all we do not have
any semblance of a consistent release pattern.  If I get my feature merged
into Traffic Control today, I have no idea when that'll actually be in a
release.  Secondly, we do not currently have any way to limit what goes
into a release besides "everything in master since the last release".  This
leads to a lot of uncertainty with new releases and lower confidence when
deploying a new release. This also forces us to not be able to move fast
when merging things into master since we are always trying to keep master
"stable".    Finally, since our releases are so big and infrequent it takes
a long time for the majority of the community to upgrade, which then
requires us to support old releases for a longer period of time.  Our
current available release is 3.0 which means there are still folks that are
on 2.2 and we should technically support it since it's n-1.  For reference,
2.2 was released in June of 2018.
I agree that not having clear objectives and definitions of good enough
lead to problems.  I don't think it's realistic to plan all of the features
that we want to put into each release, it's an open source project and
that's not how it works.  This is why I am suggesting a consistent time
based release cycle.  This will allow us to all have the same understanding
of when a release is going to be and when our features will be included in
a release.  This also allows us to have smaller releases that are more
easily digestible which allows us to hopefully move faster as a community.

BTW, 6 weeks was just a time period that felt right.  Just like with the
process if that doesn't work (too fast, too slow, etc) we can adjust that
as well.

On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <Jo...@comcast.com>
wrote:

> Did we define why/how what we've been doing isnt' working?  I see a
> consensus that it's not, suggestions on alternatives, and an ambition for
> faster and more stable; but not a definition of the thing is the reason why
> we must change?    Not saying we shouldn't, but I'm just asking so we try
> not to repeat the same mistakes following new steps.  Usually I've found
> that release blockage is derived from not defining clear objectives and
> definitions of good enough (not necessarily perfect) consistently
> throughout a release cycle.  It's not enough to start a release cycle with
> one scope of both, yet at the end continue to revise that definition.
>
> Jonathan G
>
>
> On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:
>
>     Sorry, I have taken too long to respond.
>
>     +1000000 on better CI and Testing, I think we all agree that needs to
> get
>     better and is a different beast than this topic.
>
>     As for the release branches, I am trying to follow a process similar to
>     ATS, which is usually the model we try to follow with our project.
>     While I agree that branching from master is the ideal solution, we
> tried
>     that and it didn't work.  I feel like we are trying to agree to do the
> same
>     thing that we already tried and didn't work.  I think we should have a
>     little freedom to commit things into master and, if they break things,
> work
>     on getting them fixed.  We also might commit PRs to master that are
> partial
>     implementations of a bigger feature.   If we are using a time based
>     strategy to cut our releases from master then we are going to be
>     continually worrying about making sure Master stays not broken.  I
> think we
>     should try to keep master deployable, but it's fine if it's not for
> short
>     periods of time.
>
>     I am not trying to suggest that we are "picky" about what goes into our
>     release branch.  I think we should try to cherry-pick as much as we can
>     into our release branch once it is ready to be cherry-picked.  I think
> we
>     should be able to take the tip of our release branch and deploy it
> whenever
>     we want with some confidence since we know that features/bug-fixes that
>     have been cherry-picked have already been reviewed and tested against
>     master.  Yes, its more work, but it also give us the ability to have
> more
>     confidence in what is in our release at any given time.  Again, we
> should
>     be trying to cherry-pick in everything that makes sense to cherry-pick.
>
>     I think the process that I would like to see looks like this:
>     - Cut a 4.x branch from Master
>     - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> branch
>     - Tag the 4.0.0 release from the 4.0.x branch
>     - New non-breaking changes and bug fixes that go into master are also
>     Cherry Picked to 4.x
>     - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when
> we
>     are ready for a RC.
>     - Rinse and repeat for all 4.x releases
>     - Once there's a need for a major release we make a 5.x branch from
> master
>
>     If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> then
>     4.0.x which we use to create the new 4.0.1 release (assuming we don't
> have
>     4.1 yet).  Yes, that is a more work than today, but I think A) it's not
>     really that bad and B) it gives us more control.
>
>     This should allow us to deploy much quicker and with more confidence
> than
>     is possible today.   We can also choose to deploy from any point in
> time
>     from the 4.x branch with relatively high confidence.
>
>     I'll wrap this up by saying that I don't think there's a perfect
> solution,
>     but I think we can agree on something that is worth trying at least
> with
>     the 4.x release life cycle.  Once we have some experience, we can
> figure
>     out where it is not meeting our needs and figure out how to adjust as
>     necessary.  I think we can all agree our current process is not
> working.
>
>     Thanks,
>     Dave
>
>
>
>     On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> wrote:
>
>     > +1 on more frequent releases
>     > +1 on improving our CI. We need to get our CI to the point where
> we're
>     > confident deploying changes to production if the CI passes.
>     > +1 on "trunk-based" development as opposed to GitFlow
>     > -1 on cherry-picking new features. This is risky, tedious, and
>     > represents unnecessary effort that would be better spent elsewhere,
>     > like improving our CI or adding more automated tests.
>     > +1 on cutting minor release branches directly off master
>     >
>     > I think cutting minor release branches directly off master can put us
>     > in a win-win scenario. Our releases become smaller and more frequent,
>     > and we avoid the unnecessary risk and effort of cherry-picking new
>     > features onto stale release branches.
>     >
>     > This is the process that I would propose:
>     > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
>     > 2. Develop master for 4-6 weeks.
>     > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
>     > 4. Repeat steps 2 and 3 for new minor releases as necessary.
>     > 5. Whenever a major (show-stopping) bug or security issue is found,
>     > cherry-pick the fix from master into the latest release branch (e.g.
>     > 4.1.x), and create a 4.1.1 RC.
>     >
>     > As the 4.x release manager, I really don't want to get into the
>     > business of selecting what should and should not go into a particular
>     > release. Every 4-6 weeks, we should just cut a new minor release
>     > branch off master and take everything that was contributed during
> that
>     > time, as long as everything builds fine and all the tests pass. The
>     > master branch, while it doesn't always have to be perfect, should at
>     > least always be deployable. If a new feature is merged with bugs and
>     > becomes a show-stopper, it should be either immediately remedied or
>     > reverted on master while we work on a fix.
>     >
>     > - Rawlin
>     >
>     > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
>     > <Mi...@comcast.com> wrote:
>     > >
>     > > In my opinion, in order to get to a cadence we are discussing we
> need to
>     > put a lot more work into the CI system. It has been failing
> consistently,
>     > doesn’t block PR approvals/merges and when it actually runs it does
> not
>     > test anything besides build and license headers.
>     > >
>     > > In the past couple of weeks we have had (off the top of my head):
>     > >
>     > > * Unit tests broken
>     > > * TO API tests broken
>     > > * Golang vendor issues
>     > > * TO Go build issues
>     > >
>     > > All were merged and broke master.
>     > >
>     > > Yes, having release branches sounds great and more aggressive
> cadence
>     > will minimize the amount of risk but I think that comes with the
> need to
>     > improve on our automated validation and testing.
>     > >
>     > > We have proven that we have not kept master in a good state and
> adding
>     > more releases (more branches) will make it even harder to keep that
>     > stability we are already fighting.
>     > >
>     > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> wrote:
>     > >
>     > >     Yeah, I get it. No one company should be driving release
>     > schedules/scope.
>     > >
>     > >     What I was really getting at is if ANY company has a recent
> version
>     > in a
>     > >     test or prod-like enviro (meaning the version is being
> exercised and
>     > >     thoroughly tested), then maybe we consider simply cutting a
> release
>     > from
>     > >     that version. For example, imagine company X had this version
> in a
>     > >     test/prod enviro:
>     > >
>     > >     Master-10287.7e62d07 (
>     > >
>     >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
>     > )
>     > >
>     > >     Then I would be in favor of cutting the 4.0 release from
>     > >     Master-10287.7e62d07 as we know it is proven to work based on
>     > company X's
>     > >     actual use of it.
>     > >
>     > >     The truth is (imo), we don't have the bandwidth to manually
> verify
>     > releases
>     > >     that we are not using/have not used nor do we have the
> necessary
>     > automated
>     > >     test coverage to verify these releases. So most of our
> releases are
>     > >     significantly untested.
>     > >
>     > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> ocket8888@gmail.com>
>     > wrote:
>     > >
>     > >     > I'm really not a fan of allowing Comcast to dictate the
> release
>     > scope and
>     > >     > schedule. If cherry-picking is too messy, then we can just
> cut new
>     > minor
>     > >     > releases directly from master.
>     > >     >
>     > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> mitchell852@gmail.com>
>     > wrote:
>     > >     >
>     > >     > > I don't think it's as easy as cherry picking (backporting)
>     > certain
>     > >     > features
>     > >     > > into a release branch. I could be wrong but I really don't
> think
>     > it is.
>     > >     > >
>     > >     > > So what I'm hearing is that 4.0 gets cut from master and
> we go
>     > through
>     > >     > with
>     > >     > > our normal process of testing, validating, etc. At the same
>     > time, 4.1
>     > >     > > branch is created from 4.0. Master moves on as normal.
> Let's say
>     > 25
>     > >     > commits
>     > >     > > come in to master during the next 6 weeks. During that 6
> weeks,
>     > we cherry
>     > >     > > pick only the features that look interesting for 4.1? (the
> rest
>     > just stay
>     > >     > > in master and will get picked up in 5.0?)
>     > >     > >
>     > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
>     > weeks apart
>     > >     > and
>     > >     > > have only "interesting" features so the releases are
>     > incrementally very
>     > >     > > small and much easier to test/validate, however, those
> features
>     > might
>     > >     > > depend on other commits. At some point, cherry picking
> becomes
>     > impossible
>     > >     > > as the foundation of the code base has shifted so
> drastically.
>     > Cherry
>     > >     > > picking will become very messy and introduce a high level
> of
>     > risk imo.
>     > >     > >
>     > >     > > Also, you'd have to manage the scope of these minor
> releases
>     > which we are
>     > >     > > not very good at because the designated release manager
> has a
>     > full-time
>     > >     > job
>     > >     > > to attend to and now we have even more releases to
>     > >     > > scope/test/validate/approve. I like the idea of faster
> releases
>     > but this
>     > >     > > sounds like more work that nobody has the bandwidth for.
>     > >     > >
>     > >     > > Ok, so rather than shooting down an idea with no
> alternative
>     > proposal, I
>     > >     > > propose this. At Comcast, we are trying to get to the point
>     > where we
>     > >     > > release TC to prod from the head of master every few
> weeks. What
>     > if every
>     > >     > > time Comcast does an internal upgrade, they note the
> commit hash
>     > and once
>     > >     > > comcast is happy with the upgrade in their prod enviro,
> they
>     > propose a
>     > >     > > release pinned to that commit hash? With this approach
> you'd end
>     > up with:
>     > >     > >
>     > >     > > 1. fast releases (hopefully every few weeks)
>     > >     > > 2. well tested releases as it's in a real prod enviro
>     > >     > > 3. little/no validation work for anyone
>     > >     > >
>     > >     > > Also, this doesn't have to be Comcast. If anyone else is
> out
>     > front at the
>     > >     > > edge of master and running it in a trusted enviro, they
> could
>     > propose a
>     > >     > > release as well.
>     > >     > >
>     > >     > > Jeremy
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > >
>     > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
>     > david.neuman64@gmail.com>
>     > >     > > wrote:
>     > >     > >
>     > >     > > > I have been thinking about how we can get better with our
>     > release
>     > >     > > cadence.
>     > >     > > > I feel like we have slowed to a crawl and not been as
> good as
>     > we should
>     > >     > > > about how we release.  Our last Major release was in
> March and
>     > we
>     > >     > haven't
>     > >     > > > had a real release since.   Moving forward I would like
> to see
>     > us get
>     > >     > on
>     > >     > > a
>     > >     > > > more consistent cadence and provide smaller releases
> that can
>     > be more
>     > >     > > > easily digested by the community.  It's my hope that we
> can
>     > get to a
>     > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> we are
>     > >     > releasing
>     > >     > > > in such a way that not all releases all required (e.g.
> if you
>     > are on
>     > >     > 4.0
>     > >     > > > you don't need 4.1, you can just install 4.2), and we
> are more
>     > >     > deliberate
>     > >     > > > about what we put into a release.
>     > >     > > >
>     > >     > > > I think we can accomplish this by using our release
> branches
>     > better.
>     > >     > For
>     > >     > > > each major release we will cut a release branch from
> master
>     > (e.g. 4.x)
>     > >     > > and
>     > >     > > > then we will use cherry-picking to add new features and
> bug
>     > fixes to
>     > >     > that
>     > >     > > > release.  This means that if 4.0 has been released and
> you
>     > want to get
>     > >     > > your
>     > >     > > > feature/bug fix in 4.1, you will first submit your PR to
>     > master and
>     > >     > then
>     > >     > > > cheery pick your squash merged commit to the 4.x branch
> which
>     > we will
>     > >     > use
>     > >     > > > to create the 4.1 release.  I think we should allow
> either the
>     > >     > > contributor
>     > >     > > > or the committer (who is merging the PR) to suggest if a
>     > feature goes
>     > >     > > into
>     > >     > > > the release branch, and if we disagree we can take it to
> the
>     > list.  If
>     > >     > we
>     > >     > > > decide that every PR to master also goes into 4.1 then
> so be
>     > it, but at
>     > >     > > > least we are making a conscious decision of what goes
> into the
>     > next
>     > >     > > > release.   This will allow us to not be so worried about
> what
>     > we are
>     > >     > > > merging into master and how that will affect our next
> release.
>     > >     > According
>     > >     > > > to our 6 week cadence we will cut a new release from the
>     > current
>     > >     > feature
>     > >     > > > branch and put that up for a vote.  These releases will
> be
>     > small and
>     > >     > > > testable enough that we can get feedback quickly and
> move from
>     > RC to
>     > >     > > > release in a short time.  If a release has too many
> issues
>     > then we may
>     > >     > > end
>     > >     > > > up deciding to skip that minor release in favor of the
> next
>     > one,
>     > >     > > hopefully
>     > >     > > > this does not happen very often if at all.
>     > >     > > >
>     > >     > > > Once a breaking change is introduced to master which will
>     > require a new
>     > >     > > > major release, we will create a new major release branch
> from
>     > master
>     > >     > > (which
>     > >     > > > will mean a new release manager).  We will then repeat
> the
>     > same process
>     > >     > > > with the new release branch, etc, etc.
>     > >     > > >
>     > >     > > > As for LTS we will provide support for the latest major
>     > release plus
>     > >     > the
>     > >     > > > one previous.  So, once we release 4.0 we will support
> 4.0 and
>     > 3.1.
>     > >     > Any
>     > >     > > > security issues that arise will -- if present -- be
> applied to
>     > each of
>     > >     > > > these versions.  Once 4.1 is released we will support
> 4.1 and
>     > 3.1, etc,
>     > >     > > > etc.
>     > >     > > >
>     > >     > > > Please let me know if you have any thoughts on this
> plan.  If
>     > there are
>     > >     > > no
>     > >     > > > major objections I would like to try this with 4.x with
> the
>     > idea that
>     > >     > we
>     > >     > > > will adjust how we do things as necessary.
>     > >     > > >
>     > >     > > > We need to get better at releasing and something needs to
>     > change,
>     > >     > > hopefully
>     > >     > > > this can get us going in the right direction.
>     > >     > > >
>     > >     > > > Thanks,
>     > >     > > > Dave
>     > >     > > >
>     > >     > > >
>     > >     > > > TL;DR -  I am proposing a 6 week release cycle using
>     > cherry-picks to
>     > >     > our
>     > >     > > > release branch to control what goes into a minor release.
>     > Major
>     > >     > releases
>     > >     > > > will be cut from master as necessary - such as if there
> is a
>     > breaking
>     > >     > > > change that is introduced.
>     > >     > > >
>     > >     > >
>     > >     >
>     > >
>     > >
>     >
>
>
>

Re: [EXTERNAL] Re: Release Process

["Gray, Jonathan" <Jo...@comcast.com>]

Did we define why/how what we've been doing isnt' working?  I see a consensus that it's not, suggestions on alternatives, and an ambition for faster and more stable; but not a definition of the thing is the reason why we must change?    Not saying we shouldn't, but I'm just asking so we try not to repeat the same mistakes following new steps.  Usually I've found that release blockage is derived from not defining clear objectives and definitions of good enough (not necessarily perfect) consistently throughout a release cycle.  It's not enough to start a release cycle with one scope of both, yet at the end continue to revise that definition.

Jonathan G


On 10/30/19, 2:36 PM, "Dave Neuman" <ne...@apache.org> wrote:

    Sorry, I have taken too long to respond.

    +1000000 on better CI and Testing, I think we all agree that needs to get
    better and is a different beast than this topic.

    As for the release branches, I am trying to follow a process similar to
    ATS, which is usually the model we try to follow with our project.
    While I agree that branching from master is the ideal solution, we tried
    that and it didn't work.  I feel like we are trying to agree to do the same
    thing that we already tried and didn't work.  I think we should have a
    little freedom to commit things into master and, if they break things, work
    on getting them fixed.  We also might commit PRs to master that are partial
    implementations of a bigger feature.   If we are using a time based
    strategy to cut our releases from master then we are going to be
    continually worrying about making sure Master stays not broken.  I think we
    should try to keep master deployable, but it's fine if it's not for short
    periods of time.

    I am not trying to suggest that we are "picky" about what goes into our
    release branch.  I think we should try to cherry-pick as much as we can
    into our release branch once it is ready to be cherry-picked.  I think we
    should be able to take the tip of our release branch and deploy it whenever
    we want with some confidence since we know that features/bug-fixes that
    have been cherry-picked have already been reviewed and tested against
    master.  Yes, its more work, but it also give us the ability to have more
    confidence in what is in our release at any given time.  Again, we should
    be trying to cherry-pick in everything that makes sense to cherry-pick.

    I think the process that I would like to see looks like this:
    - Cut a 4.x branch from Master
    - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x branch
    - Tag the 4.0.0 release from the 4.0.x branch
    - New non-breaking changes and bug fixes that go into master are also
    Cherry Picked to 4.x
    - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
    are ready for a RC.
    - Rinse and repeat for all 4.x releases
    - Once there's a need for a major release we make a 5.x branch from master

    If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, then
    4.0.x which we use to create the new 4.0.1 release (assuming we don't have
    4.1 yet).  Yes, that is a more work than today, but I think A) it's not
    really that bad and B) it gives us more control.

    This should allow us to deploy much quicker and with more confidence than
    is possible today.   We can also choose to deploy from any point in time
    from the 4.x branch with relatively high confidence.

    I'll wrap this up by saying that I don't think there's a perfect solution,
    but I think we can agree on something that is worth trying at least with
    the 4.x release life cycle.  Once we have some experience, we can figure
    out where it is not meeting our needs and figure out how to adjust as
    necessary.  I think we can all agree our current process is not working.

    Thanks,
    Dave



    On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org> wrote:

    > +1 on more frequent releases
    > +1 on improving our CI. We need to get our CI to the point where we're
    > confident deploying changes to production if the CI passes.
    > +1 on "trunk-based" development as opposed to GitFlow
    > -1 on cherry-picking new features. This is risky, tedious, and
    > represents unnecessary effort that would be better spent elsewhere,
    > like improving our CI or adding more automated tests.
    > +1 on cutting minor release branches directly off master
    >
    > I think cutting minor release branches directly off master can put us
    > in a win-win scenario. Our releases become smaller and more frequent,
    > and we avoid the unnecessary risk and effort of cherry-picking new
    > features onto stale release branches.
    >
    > This is the process that I would propose:
    > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
    > 2. Develop master for 4-6 weeks.
    > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
    > 4. Repeat steps 2 and 3 for new minor releases as necessary.
    > 5. Whenever a major (show-stopping) bug or security issue is found,
    > cherry-pick the fix from master into the latest release branch (e.g.
    > 4.1.x), and create a 4.1.1 RC.
    >
    > As the 4.x release manager, I really don't want to get into the
    > business of selecting what should and should not go into a particular
    > release. Every 4-6 weeks, we should just cut a new minor release
    > branch off master and take everything that was contributed during that
    > time, as long as everything builds fine and all the tests pass. The
    > master branch, while it doesn't always have to be perfect, should at
    > least always be deployable. If a new feature is merged with bugs and
    > becomes a show-stopper, it should be either immediately remedied or
    > reverted on master while we work on a fix.
    >
    > - Rawlin
    >
    > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
    > <Mi...@comcast.com> wrote:
    > >
    > > In my opinion, in order to get to a cadence we are discussing we need to
    > put a lot more work into the CI system. It has been failing consistently,
    > doesn’t block PR approvals/merges and when it actually runs it does not
    > test anything besides build and license headers.
    > >
    > > In the past couple of weeks we have had (off the top of my head):
    > >
    > > * Unit tests broken
    > > * TO API tests broken
    > > * Golang vendor issues
    > > * TO Go build issues
    > >
    > > All were merged and broke master.
    > >
    > > Yes, having release branches sounds great and more aggressive cadence
    > will minimize the amount of risk but I think that comes with the need to
    > improve on our automated validation and testing.
    > >
    > > We have proven that we have not kept master in a good state and adding
    > more releases (more branches) will make it even harder to keep that
    > stability we are already fighting.
    > >
    > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
    > >
    > >     Yeah, I get it. No one company should be driving release
    > schedules/scope.
    > >
    > >     What I was really getting at is if ANY company has a recent version
    > in a
    > >     test or prod-like enviro (meaning the version is being exercised and
    > >     thoroughly tested), then maybe we consider simply cutting a release
    > from
    > >     that version. For example, imagine company X had this version in a
    > >     test/prod enviro:
    > >
    > >     Master-10287.7e62d07 (
    > >
    > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
    > )
    > >
    > >     Then I would be in favor of cutting the 4.0 release from
    > >     Master-10287.7e62d07 as we know it is proven to work based on
    > company X's
    > >     actual use of it.
    > >
    > >     The truth is (imo), we don't have the bandwidth to manually verify
    > releases
    > >     that we are not using/have not used nor do we have the necessary
    > automated
    > >     test coverage to verify these releases. So most of our releases are
    > >     significantly untested.
    > >
    > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
    > wrote:
    > >
    > >     > I'm really not a fan of allowing Comcast to dictate the release
    > scope and
    > >     > schedule. If cherry-picking is too messy, then we can just cut new
    > minor
    > >     > releases directly from master.
    > >     >
    > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com>
    > wrote:
    > >     >
    > >     > > I don't think it's as easy as cherry picking (backporting)
    > certain
    > >     > features
    > >     > > into a release branch. I could be wrong but I really don't think
    > it is.
    > >     > >
    > >     > > So what I'm hearing is that 4.0 gets cut from master and we go
    > through
    > >     > with
    > >     > > our normal process of testing, validating, etc. At the same
    > time, 4.1
    > >     > > branch is created from 4.0. Master moves on as normal. Let's say
    > 25
    > >     > commits
    > >     > > come in to master during the next 6 weeks. During that 6 weeks,
    > we cherry
    > >     > > pick only the features that look interesting for 4.1? (the rest
    > just stay
    > >     > > in master and will get picked up in 5.0?)
    > >     > >
    > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
    > weeks apart
    > >     > and
    > >     > > have only "interesting" features so the releases are
    > incrementally very
    > >     > > small and much easier to test/validate, however, those features
    > might
    > >     > > depend on other commits. At some point, cherry picking becomes
    > impossible
    > >     > > as the foundation of the code base has shifted so drastically.
    > Cherry
    > >     > > picking will become very messy and introduce a high level of
    > risk imo.
    > >     > >
    > >     > > Also, you'd have to manage the scope of these minor releases
    > which we are
    > >     > > not very good at because the designated release manager has a
    > full-time
    > >     > job
    > >     > > to attend to and now we have even more releases to
    > >     > > scope/test/validate/approve. I like the idea of faster releases
    > but this
    > >     > > sounds like more work that nobody has the bandwidth for.
    > >     > >
    > >     > > Ok, so rather than shooting down an idea with no alternative
    > proposal, I
    > >     > > propose this. At Comcast, we are trying to get to the point
    > where we
    > >     > > release TC to prod from the head of master every few weeks. What
    > if every
    > >     > > time Comcast does an internal upgrade, they note the commit hash
    > and once
    > >     > > comcast is happy with the upgrade in their prod enviro, they
    > propose a
    > >     > > release pinned to that commit hash? With this approach you'd end
    > up with:
    > >     > >
    > >     > > 1. fast releases (hopefully every few weeks)
    > >     > > 2. well tested releases as it's in a real prod enviro
    > >     > > 3. little/no validation work for anyone
    > >     > >
    > >     > > Also, this doesn't have to be Comcast. If anyone else is out
    > front at the
    > >     > > edge of master and running it in a trusted enviro, they could
    > propose a
    > >     > > release as well.
    > >     > >
    > >     > > Jeremy
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > >
    > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
    > david.neuman64@gmail.com>
    > >     > > wrote:
    > >     > >
    > >     > > > I have been thinking about how we can get better with our
    > release
    > >     > > cadence.
    > >     > > > I feel like we have slowed to a crawl and not been as good as
    > we should
    > >     > > > about how we release.  Our last Major release was in March and
    > we
    > >     > haven't
    > >     > > > had a real release since.   Moving forward I would like to see
    > us get
    > >     > on
    > >     > > a
    > >     > > > more consistent cadence and provide smaller releases that can
    > be more
    > >     > > > easily digested by the community.  It's my hope that we can
    > get to a
    > >     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
    > >     > releasing
    > >     > > > in such a way that not all releases all required (e.g. if you
    > are on
    > >     > 4.0
    > >     > > > you don't need 4.1, you can just install 4.2), and we are more
    > >     > deliberate
    > >     > > > about what we put into a release.
    > >     > > >
    > >     > > > I think we can accomplish this by using our release branches
    > better.
    > >     > For
    > >     > > > each major release we will cut a release branch from master
    > (e.g. 4.x)
    > >     > > and
    > >     > > > then we will use cherry-picking to add new features and bug
    > fixes to
    > >     > that
    > >     > > > release.  This means that if 4.0 has been released and you
    > want to get
    > >     > > your
    > >     > > > feature/bug fix in 4.1, you will first submit your PR to
    > master and
    > >     > then
    > >     > > > cheery pick your squash merged commit to the 4.x branch which
    > we will
    > >     > use
    > >     > > > to create the 4.1 release.  I think we should allow either the
    > >     > > contributor
    > >     > > > or the committer (who is merging the PR) to suggest if a
    > feature goes
    > >     > > into
    > >     > > > the release branch, and if we disagree we can take it to the
    > list.  If
    > >     > we
    > >     > > > decide that every PR to master also goes into 4.1 then so be
    > it, but at
    > >     > > > least we are making a conscious decision of what goes into the
    > next
    > >     > > > release.   This will allow us to not be so worried about what
    > we are
    > >     > > > merging into master and how that will affect our next release.
    > >     > According
    > >     > > > to our 6 week cadence we will cut a new release from the
    > current
    > >     > feature
    > >     > > > branch and put that up for a vote.  These releases will be
    > small and
    > >     > > > testable enough that we can get feedback quickly and move from
    > RC to
    > >     > > > release in a short time.  If a release has too many issues
    > then we may
    > >     > > end
    > >     > > > up deciding to skip that minor release in favor of the next
    > one,
    > >     > > hopefully
    > >     > > > this does not happen very often if at all.
    > >     > > >
    > >     > > > Once a breaking change is introduced to master which will
    > require a new
    > >     > > > major release, we will create a new major release branch from
    > master
    > >     > > (which
    > >     > > > will mean a new release manager).  We will then repeat the
    > same process
    > >     > > > with the new release branch, etc, etc.
    > >     > > >
    > >     > > > As for LTS we will provide support for the latest major
    > release plus
    > >     > the
    > >     > > > one previous.  So, once we release 4.0 we will support 4.0 and
    > 3.1.
    > >     > Any
    > >     > > > security issues that arise will -- if present -- be applied to
    > each of
    > >     > > > these versions.  Once 4.1 is released we will support 4.1 and
    > 3.1, etc,
    > >     > > > etc.
    > >     > > >
    > >     > > > Please let me know if you have any thoughts on this plan.  If
    > there are
    > >     > > no
    > >     > > > major objections I would like to try this with 4.x with the
    > idea that
    > >     > we
    > >     > > > will adjust how we do things as necessary.
    > >     > > >
    > >     > > > We need to get better at releasing and something needs to
    > change,
    > >     > > hopefully
    > >     > > > this can get us going in the right direction.
    > >     > > >
    > >     > > > Thanks,
    > >     > > > Dave
    > >     > > >
    > >     > > >
    > >     > > > TL;DR -  I am proposing a 6 week release cycle using
    > cherry-picks to
    > >     > our
    > >     > > > release branch to control what goes into a minor release.
    > Major
    > >     > releases
    > >     > > > will be cut from master as necessary - such as if there is a
    > breaking
    > >     > > > change that is introduced.
    > >     > > >
    > >     > >
    > >     >
    > >
    > >
    >

Re: [EXTERNAL] Re: Release Process

[Dave Neuman <ne...@apache.org>]

I was trying to avoid using that term since it is different enough to not
be exactly the same, though it is very similar.

On Wed, Oct 30, 2019 at 7:34 PM Gray, Jonathan <Jo...@comcast.com>
wrote:

> It sounds like you're wanting to convert from Github Flow to regular
> Gitflow, but change what branches do what.
> https://lucamezzalira.com/2014/03/10/git-flow-vs-github-flow/
>
> Jonathan G
>
> On 10/30/19, 5:35 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
>     I guess my concern is really about what possible reasons we might have
>     to not cherry-pick a specific PR that had been merged into master
>     already. The only reasons I can think of to not cherry-pick something
>     would be:
>     1. the PR was completely broken (e.g. breaks the build, tests don't
>     pass, doesn't compile, etc).
>     2. the PR one one of a larger multi-PR "feature", and *we know*
>     releasing some parts early could cause problems (that knowing part is
>     crucial)
>
>     We should always strive to catch issue #1 in our CI before it merges
>     into master. When things like that merge it can block people from
>     doing development starting at that point until it's fixed.
>     I can see why issue #2 might arise due to multiple different people
>     developing one larger feature, but we should still strive to break
>     larger features up into individual pieces that can reasonably stand on
>     their own if possible. Due to time constraints or other reasons I can
>     see that might be a little difficult.
>
>     That said, I would still strive to make the 4.x release branch a
>     snapshot of master at some "known good point" in time. Let's say we
>     cut 4.x at commit A, then a broken commit B gets merged into master.
>     If it's completely obvious that it's a broken commit, I'd wait until
>     at least a commit C gets merged to fix it. Then I would cherry-pick B
>     and C into 4.x at the same time. If the fix is really like a commit E
>     with unrelated commits C and D in between, ideally I would still wait
>     to cherry-pick C and D until a "fix" commit E was merged (to fix
>     commit B), at which point I'd cherrypick commits B through E at the
>     same time, in order, to avoid any possible merge conflicts.
>
>     I think to make that process easier for myself I would just
>     cherry-pick everything into the 4.x branch as it gets merged into
>     master. Then, after 4-6 weeks of cherry-picking commits, if unstable
>     commits were not fixed/completed in time, I would reset the 4.x branch
>     to the last stable commit. This would mean that unstable
>     commits/incomplete features could potentially block things from going
>     into the release. I might count that as a positive because it would
>     incentivize us to improve our CI and do our best to also keep the
>     master branch as stable as possible.
>
>     The end result would be the same as just continually jumping up the
>     master branch from known good, stable points in time to the next,
>     creating a minor release branch at each of those points.
>
>     - Rawlin
>
>
>
>     On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
>     >
>     > Sorry, I have taken too long to respond.
>     >
>     > +1000000 on better CI and Testing, I think we all agree that needs
> to get
>     > better and is a different beast than this topic.
>     >
>     > As for the release branches, I am trying to follow a process similar
> to
>     > ATS, which is usually the model we try to follow with our project.
>     > While I agree that branching from master is the ideal solution, we
> tried
>     > that and it didn't work.  I feel like we are trying to agree to do
> the same
>     > thing that we already tried and didn't work.  I think we should have
> a
>     > little freedom to commit things into master and, if they break
> things, work
>     > on getting them fixed.  We also might commit PRs to master that are
> partial
>     > implementations of a bigger feature.   If we are using a time based
>     > strategy to cut our releases from master then we are going to be
>     > continually worrying about making sure Master stays not broken.  I
> think we
>     > should try to keep master deployable, but it's fine if it's not for
> short
>     > periods of time.
>     >
>     > I am not trying to suggest that we are "picky" about what goes into
> our
>     > release branch.  I think we should try to cherry-pick as much as we
> can
>     > into our release branch once it is ready to be cherry-picked.  I
> think we
>     > should be able to take the tip of our release branch and deploy it
> whenever
>     > we want with some confidence since we know that features/bug-fixes
> that
>     > have been cherry-picked have already been reviewed and tested against
>     > master.  Yes, its more work, but it also give us the ability to have
> more
>     > confidence in what is in our release at any given time.  Again, we
> should
>     > be trying to cherry-pick in everything that makes sense to
> cherry-pick.
>     >
>     > I think the process that I would like to see looks like this:
>     > - Cut a 4.x branch from Master
>     > - Once we think we are ready for a RC cut a 4.0.x branch from the
> 4.x branch
>     > - Tag the 4.0.0 release from the 4.0.x branch
>     > - New non-breaking changes and bug fixes that go into master are also
>     > Cherry Picked to 4.x
>     > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag
> when we
>     > are ready for a RC.
>     > - Rinse and repeat for all 4.x releases
>     > - Once there's a need for a major release we make a 5.x branch from
> master
>     >
>     > If we find a bug in 4.0.0 then we commit the fix to master, then
> 4.x, then
>     > 4.0.x which we use to create the new 4.0.1 release (assuming we
> don't have
>     > 4.1 yet).  Yes, that is a more work than today, but I think A) it's
> not
>     > really that bad and B) it gives us more control.
>     >
>     > This should allow us to deploy much quicker and with more confidence
> than
>     > is possible today.   We can also choose to deploy from any point in
> time
>     > from the 4.x branch with relatively high confidence.
>     >
>     > I'll wrap this up by saying that I don't think there's a perfect
> solution,
>     > but I think we can agree on something that is worth trying at least
> with
>     > the 4.x release life cycle.  Once we have some experience, we can
> figure
>     > out where it is not meeting our needs and figure out how to adjust as
>     > necessary.  I think we can all agree our current process is not
> working.
>     >
>     > Thanks,
>     > Dave
>     >
>     >
>     >
>     > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> wrote:
>     >
>     > > +1 on more frequent releases
>     > > +1 on improving our CI. We need to get our CI to the point where
> we're
>     > > confident deploying changes to production if the CI passes.
>     > > +1 on "trunk-based" development as opposed to GitFlow
>     > > -1 on cherry-picking new features. This is risky, tedious, and
>     > > represents unnecessary effort that would be better spent elsewhere,
>     > > like improving our CI or adding more automated tests.
>     > > +1 on cutting minor release branches directly off master
>     > >
>     > > I think cutting minor release branches directly off master can put
> us
>     > > in a win-win scenario. Our releases become smaller and more
> frequent,
>     > > and we avoid the unnecessary risk and effort of cherry-picking new
>     > > features onto stale release branches.
>     > >
>     > > This is the process that I would propose:
>     > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
>     > > 2. Develop master for 4-6 weeks.
>     > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
>     > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
>     > > 5. Whenever a major (show-stopping) bug or security issue is found,
>     > > cherry-pick the fix from master into the latest release branch
> (e.g.
>     > > 4.1.x), and create a 4.1.1 RC.
>     > >
>     > > As the 4.x release manager, I really don't want to get into the
>     > > business of selecting what should and should not go into a
> particular
>     > > release. Every 4-6 weeks, we should just cut a new minor release
>     > > branch off master and take everything that was contributed during
> that
>     > > time, as long as everything builds fine and all the tests pass. The
>     > > master branch, while it doesn't always have to be perfect, should
> at
>     > > least always be deployable. If a new feature is merged with bugs
> and
>     > > becomes a show-stopper, it should be either immediately remedied or
>     > > reverted on master while we work on a fix.
>     > >
>     > > - Rawlin
>     > >
>     > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
>     > > <Mi...@comcast.com> wrote:
>     > > >
>     > > > In my opinion, in order to get to a cadence we are discussing we
> need to
>     > > put a lot more work into the CI system. It has been failing
> consistently,
>     > > doesn’t block PR approvals/merges and when it actually runs it
> does not
>     > > test anything besides build and license headers.
>     > > >
>     > > > In the past couple of weeks we have had (off the top of my head):
>     > > >
>     > > > * Unit tests broken
>     > > > * TO API tests broken
>     > > > * Golang vendor issues
>     > > > * TO Go build issues
>     > > >
>     > > > All were merged and broke master.
>     > > >
>     > > > Yes, having release branches sounds great and more aggressive
> cadence
>     > > will minimize the amount of risk but I think that comes with the
> need to
>     > > improve on our automated validation and testing.
>     > > >
>     > > > We have proven that we have not kept master in a good state and
> adding
>     > > more releases (more branches) will make it even harder to keep that
>     > > stability we are already fighting.
>     > > >
>     > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> wrote:
>     > > >
>     > > >     Yeah, I get it. No one company should be driving release
>     > > schedules/scope.
>     > > >
>     > > >     What I was really getting at is if ANY company has a recent
> version
>     > > in a
>     > > >     test or prod-like enviro (meaning the version is being
> exercised and
>     > > >     thoroughly tested), then maybe we consider simply cutting a
> release
>     > > from
>     > > >     that version. For example, imagine company X had this
> version in a
>     > > >     test/prod enviro:
>     > > >
>     > > >     Master-10287.7e62d07 (
>     > > >
>     > >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
>     > > )
>     > > >
>     > > >     Then I would be in favor of cutting the 4.0 release from
>     > > >     Master-10287.7e62d07 as we know it is proven to work based on
>     > > company X's
>     > > >     actual use of it.
>     > > >
>     > > >     The truth is (imo), we don't have the bandwidth to manually
> verify
>     > > releases
>     > > >     that we are not using/have not used nor do we have the
> necessary
>     > > automated
>     > > >     test coverage to verify these releases. So most of our
> releases are
>     > > >     significantly untested.
>     > > >
>     > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> ocket8888@gmail.com>
>     > > wrote:
>     > > >
>     > > >     > I'm really not a fan of allowing Comcast to dictate the
> release
>     > > scope and
>     > > >     > schedule. If cherry-picking is too messy, then we can just
> cut new
>     > > minor
>     > > >     > releases directly from master.
>     > > >     >
>     > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> mitchell852@gmail.com>
>     > > wrote:
>     > > >     >
>     > > >     > > I don't think it's as easy as cherry picking
> (backporting)
>     > > certain
>     > > >     > features
>     > > >     > > into a release branch. I could be wrong but I really
> don't think
>     > > it is.
>     > > >     > >
>     > > >     > > So what I'm hearing is that 4.0 gets cut from master and
> we go
>     > > through
>     > > >     > with
>     > > >     > > our normal process of testing, validating, etc. At the
> same
>     > > time, 4.1
>     > > >     > > branch is created from 4.0. Master moves on as normal.
> Let's say
>     > > 25
>     > > >     > commits
>     > > >     > > come in to master during the next 6 weeks. During that 6
> weeks,
>     > > we cherry
>     > > >     > > pick only the features that look interesting for 4.1?
> (the rest
>     > > just stay
>     > > >     > > in master and will get picked up in 5.0?)
>     > > >     > >
>     > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> roughly 6
>     > > weeks apart
>     > > >     > and
>     > > >     > > have only "interesting" features so the releases are
>     > > incrementally very
>     > > >     > > small and much easier to test/validate, however, those
> features
>     > > might
>     > > >     > > depend on other commits. At some point, cherry picking
> becomes
>     > > impossible
>     > > >     > > as the foundation of the code base has shifted so
> drastically.
>     > > Cherry
>     > > >     > > picking will become very messy and introduce a high
> level of
>     > > risk imo.
>     > > >     > >
>     > > >     > > Also, you'd have to manage the scope of these minor
> releases
>     > > which we are
>     > > >     > > not very good at because the designated release manager
> has a
>     > > full-time
>     > > >     > job
>     > > >     > > to attend to and now we have even more releases to
>     > > >     > > scope/test/validate/approve. I like the idea of faster
> releases
>     > > but this
>     > > >     > > sounds like more work that nobody has the bandwidth for.
>     > > >     > >
>     > > >     > > Ok, so rather than shooting down an idea with no
> alternative
>     > > proposal, I
>     > > >     > > propose this. At Comcast, we are trying to get to the
> point
>     > > where we
>     > > >     > > release TC to prod from the head of master every few
> weeks. What
>     > > if every
>     > > >     > > time Comcast does an internal upgrade, they note the
> commit hash
>     > > and once
>     > > >     > > comcast is happy with the upgrade in their prod enviro,
> they
>     > > propose a
>     > > >     > > release pinned to that commit hash? With this approach
> you'd end
>     > > up with:
>     > > >     > >
>     > > >     > > 1. fast releases (hopefully every few weeks)
>     > > >     > > 2. well tested releases as it's in a real prod enviro
>     > > >     > > 3. little/no validation work for anyone
>     > > >     > >
>     > > >     > > Also, this doesn't have to be Comcast. If anyone else is
> out
>     > > front at the
>     > > >     > > edge of master and running it in a trusted enviro, they
> could
>     > > propose a
>     > > >     > > release as well.
>     > > >     > >
>     > > >     > > Jeremy
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > >
>     > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
>     > > david.neuman64@gmail.com>
>     > > >     > > wrote:
>     > > >     > >
>     > > >     > > > I have been thinking about how we can get better with
> our
>     > > release
>     > > >     > > cadence.
>     > > >     > > > I feel like we have slowed to a crawl and not been as
> good as
>     > > we should
>     > > >     > > > about how we release.  Our last Major release was in
> March and
>     > > we
>     > > >     > haven't
>     > > >     > > > had a real release since.   Moving forward I would
> like to see
>     > > us get
>     > > >     > on
>     > > >     > > a
>     > > >     > > > more consistent cadence and provide smaller releases
> that can
>     > > be more
>     > > >     > > > easily digested by the community.  It's my hope that
> we can
>     > > get to a
>     > > >     > > > cadence where we are doing releases every 4 to 6
> weeks,  we are
>     > > >     > releasing
>     > > >     > > > in such a way that not all releases all required (e.g.
> if you
>     > > are on
>     > > >     > 4.0
>     > > >     > > > you don't need 4.1, you can just install 4.2), and we
> are more
>     > > >     > deliberate
>     > > >     > > > about what we put into a release.
>     > > >     > > >
>     > > >     > > > I think we can accomplish this by using our release
> branches
>     > > better.
>     > > >     > For
>     > > >     > > > each major release we will cut a release branch from
> master
>     > > (e.g. 4.x)
>     > > >     > > and
>     > > >     > > > then we will use cherry-picking to add new features
> and bug
>     > > fixes to
>     > > >     > that
>     > > >     > > > release.  This means that if 4.0 has been released and
> you
>     > > want to get
>     > > >     > > your
>     > > >     > > > feature/bug fix in 4.1, you will first submit your PR
> to
>     > > master and
>     > > >     > then
>     > > >     > > > cheery pick your squash merged commit to the 4.x
> branch which
>     > > we will
>     > > >     > use
>     > > >     > > > to create the 4.1 release.  I think we should allow
> either the
>     > > >     > > contributor
>     > > >     > > > or the committer (who is merging the PR) to suggest if
> a
>     > > feature goes
>     > > >     > > into
>     > > >     > > > the release branch, and if we disagree we can take it
> to the
>     > > list.  If
>     > > >     > we
>     > > >     > > > decide that every PR to master also goes into 4.1 then
> so be
>     > > it, but at
>     > > >     > > > least we are making a conscious decision of what goes
> into the
>     > > next
>     > > >     > > > release.   This will allow us to not be so worried
> about what
>     > > we are
>     > > >     > > > merging into master and how that will affect our next
> release.
>     > > >     > According
>     > > >     > > > to our 6 week cadence we will cut a new release from
> the
>     > > current
>     > > >     > feature
>     > > >     > > > branch and put that up for a vote.  These releases
> will be
>     > > small and
>     > > >     > > > testable enough that we can get feedback quickly and
> move from
>     > > RC to
>     > > >     > > > release in a short time.  If a release has too many
> issues
>     > > then we may
>     > > >     > > end
>     > > >     > > > up deciding to skip that minor release in favor of the
> next
>     > > one,
>     > > >     > > hopefully
>     > > >     > > > this does not happen very often if at all.
>     > > >     > > >
>     > > >     > > > Once a breaking change is introduced to master which
> will
>     > > require a new
>     > > >     > > > major release, we will create a new major release
> branch from
>     > > master
>     > > >     > > (which
>     > > >     > > > will mean a new release manager).  We will then repeat
> the
>     > > same process
>     > > >     > > > with the new release branch, etc, etc.
>     > > >     > > >
>     > > >     > > > As for LTS we will provide support for the latest major
>     > > release plus
>     > > >     > the
>     > > >     > > > one previous.  So, once we release 4.0 we will support
> 4.0 and
>     > > 3.1.
>     > > >     > Any
>     > > >     > > > security issues that arise will -- if present -- be
> applied to
>     > > each of
>     > > >     > > > these versions.  Once 4.1 is released we will support
> 4.1 and
>     > > 3.1, etc,
>     > > >     > > > etc.
>     > > >     > > >
>     > > >     > > > Please let me know if you have any thoughts on this
> plan.  If
>     > > there are
>     > > >     > > no
>     > > >     > > > major objections I would like to try this with 4.x
> with the
>     > > idea that
>     > > >     > we
>     > > >     > > > will adjust how we do things as necessary.
>     > > >     > > >
>     > > >     > > > We need to get better at releasing and something needs
> to
>     > > change,
>     > > >     > > hopefully
>     > > >     > > > this can get us going in the right direction.
>     > > >     > > >
>     > > >     > > > Thanks,
>     > > >     > > > Dave
>     > > >     > > >
>     > > >     > > >
>     > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
>     > > cherry-picks to
>     > > >     > our
>     > > >     > > > release branch to control what goes into a minor
> release.
>     > > Major
>     > > >     > releases
>     > > >     > > > will be cut from master as necessary - such as if
> there is a
>     > > breaking
>     > > >     > > > change that is introduced.
>     > > >     > > >
>     > > >     > >
>     > > >     >
>     > > >
>     > > >
>     > >
>
>
>

Re: [EXTERNAL] Re: Release Process

["Gray, Jonathan" <Jo...@comcast.com>]

It sounds like you're wanting to convert from Github Flow to regular Gitflow, but change what branches do what. https://lucamezzalira.com/2014/03/10/git-flow-vs-github-flow/

Jonathan G

On 10/30/19, 5:35 PM, "Rawlin Peters" <ra...@apache.org> wrote:

    I guess my concern is really about what possible reasons we might have
    to not cherry-pick a specific PR that had been merged into master
    already. The only reasons I can think of to not cherry-pick something
    would be:
    1. the PR was completely broken (e.g. breaks the build, tests don't
    pass, doesn't compile, etc).
    2. the PR one one of a larger multi-PR "feature", and *we know*
    releasing some parts early could cause problems (that knowing part is
    crucial)

    We should always strive to catch issue #1 in our CI before it merges
    into master. When things like that merge it can block people from
    doing development starting at that point until it's fixed.
    I can see why issue #2 might arise due to multiple different people
    developing one larger feature, but we should still strive to break
    larger features up into individual pieces that can reasonably stand on
    their own if possible. Due to time constraints or other reasons I can
    see that might be a little difficult.

    That said, I would still strive to make the 4.x release branch a
    snapshot of master at some "known good point" in time. Let's say we
    cut 4.x at commit A, then a broken commit B gets merged into master.
    If it's completely obvious that it's a broken commit, I'd wait until
    at least a commit C gets merged to fix it. Then I would cherry-pick B
    and C into 4.x at the same time. If the fix is really like a commit E
    with unrelated commits C and D in between, ideally I would still wait
    to cherry-pick C and D until a "fix" commit E was merged (to fix
    commit B), at which point I'd cherrypick commits B through E at the
    same time, in order, to avoid any possible merge conflicts.

    I think to make that process easier for myself I would just
    cherry-pick everything into the 4.x branch as it gets merged into
    master. Then, after 4-6 weeks of cherry-picking commits, if unstable
    commits were not fixed/completed in time, I would reset the 4.x branch
    to the last stable commit. This would mean that unstable
    commits/incomplete features could potentially block things from going
    into the release. I might count that as a positive because it would
    incentivize us to improve our CI and do our best to also keep the
    master branch as stable as possible.

    The end result would be the same as just continually jumping up the
    master branch from known good, stable points in time to the next,
    creating a minor release branch at each of those points.

    - Rawlin



    On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
    >
    > Sorry, I have taken too long to respond.
    >
    > +1000000 on better CI and Testing, I think we all agree that needs to get
    > better and is a different beast than this topic.
    >
    > As for the release branches, I am trying to follow a process similar to
    > ATS, which is usually the model we try to follow with our project.
    > While I agree that branching from master is the ideal solution, we tried
    > that and it didn't work.  I feel like we are trying to agree to do the same
    > thing that we already tried and didn't work.  I think we should have a
    > little freedom to commit things into master and, if they break things, work
    > on getting them fixed.  We also might commit PRs to master that are partial
    > implementations of a bigger feature.   If we are using a time based
    > strategy to cut our releases from master then we are going to be
    > continually worrying about making sure Master stays not broken.  I think we
    > should try to keep master deployable, but it's fine if it's not for short
    > periods of time.
    >
    > I am not trying to suggest that we are "picky" about what goes into our
    > release branch.  I think we should try to cherry-pick as much as we can
    > into our release branch once it is ready to be cherry-picked.  I think we
    > should be able to take the tip of our release branch and deploy it whenever
    > we want with some confidence since we know that features/bug-fixes that
    > have been cherry-picked have already been reviewed and tested against
    > master.  Yes, its more work, but it also give us the ability to have more
    > confidence in what is in our release at any given time.  Again, we should
    > be trying to cherry-pick in everything that makes sense to cherry-pick.
    >
    > I think the process that I would like to see looks like this:
    > - Cut a 4.x branch from Master
    > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x branch
    > - Tag the 4.0.0 release from the 4.0.x branch
    > - New non-breaking changes and bug fixes that go into master are also
    > Cherry Picked to 4.x
    > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
    > are ready for a RC.
    > - Rinse and repeat for all 4.x releases
    > - Once there's a need for a major release we make a 5.x branch from master
    >
    > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, then
    > 4.0.x which we use to create the new 4.0.1 release (assuming we don't have
    > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
    > really that bad and B) it gives us more control.
    >
    > This should allow us to deploy much quicker and with more confidence than
    > is possible today.   We can also choose to deploy from any point in time
    > from the 4.x branch with relatively high confidence.
    >
    > I'll wrap this up by saying that I don't think there's a perfect solution,
    > but I think we can agree on something that is worth trying at least with
    > the 4.x release life cycle.  Once we have some experience, we can figure
    > out where it is not meeting our needs and figure out how to adjust as
    > necessary.  I think we can all agree our current process is not working.
    >
    > Thanks,
    > Dave
    >
    >
    >
    > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org> wrote:
    >
    > > +1 on more frequent releases
    > > +1 on improving our CI. We need to get our CI to the point where we're
    > > confident deploying changes to production if the CI passes.
    > > +1 on "trunk-based" development as opposed to GitFlow
    > > -1 on cherry-picking new features. This is risky, tedious, and
    > > represents unnecessary effort that would be better spent elsewhere,
    > > like improving our CI or adding more automated tests.
    > > +1 on cutting minor release branches directly off master
    > >
    > > I think cutting minor release branches directly off master can put us
    > > in a win-win scenario. Our releases become smaller and more frequent,
    > > and we avoid the unnecessary risk and effort of cherry-picking new
    > > features onto stale release branches.
    > >
    > > This is the process that I would propose:
    > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
    > > 2. Develop master for 4-6 weeks.
    > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
    > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
    > > 5. Whenever a major (show-stopping) bug or security issue is found,
    > > cherry-pick the fix from master into the latest release branch (e.g.
    > > 4.1.x), and create a 4.1.1 RC.
    > >
    > > As the 4.x release manager, I really don't want to get into the
    > > business of selecting what should and should not go into a particular
    > > release. Every 4-6 weeks, we should just cut a new minor release
    > > branch off master and take everything that was contributed during that
    > > time, as long as everything builds fine and all the tests pass. The
    > > master branch, while it doesn't always have to be perfect, should at
    > > least always be deployable. If a new feature is merged with bugs and
    > > becomes a show-stopper, it should be either immediately remedied or
    > > reverted on master while we work on a fix.
    > >
    > > - Rawlin
    > >
    > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
    > > <Mi...@comcast.com> wrote:
    > > >
    > > > In my opinion, in order to get to a cadence we are discussing we need to
    > > put a lot more work into the CI system. It has been failing consistently,
    > > doesn’t block PR approvals/merges and when it actually runs it does not
    > > test anything besides build and license headers.
    > > >
    > > > In the past couple of weeks we have had (off the top of my head):
    > > >
    > > > * Unit tests broken
    > > > * TO API tests broken
    > > > * Golang vendor issues
    > > > * TO Go build issues
    > > >
    > > > All were merged and broke master.
    > > >
    > > > Yes, having release branches sounds great and more aggressive cadence
    > > will minimize the amount of risk but I think that comes with the need to
    > > improve on our automated validation and testing.
    > > >
    > > > We have proven that we have not kept master in a good state and adding
    > > more releases (more branches) will make it even harder to keep that
    > > stability we are already fighting.
    > > >
    > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
    > > >
    > > >     Yeah, I get it. No one company should be driving release
    > > schedules/scope.
    > > >
    > > >     What I was really getting at is if ANY company has a recent version
    > > in a
    > > >     test or prod-like enviro (meaning the version is being exercised and
    > > >     thoroughly tested), then maybe we consider simply cutting a release
    > > from
    > > >     that version. For example, imagine company X had this version in a
    > > >     test/prod enviro:
    > > >
    > > >     Master-10287.7e62d07 (
    > > >
    > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
    > > )
    > > >
    > > >     Then I would be in favor of cutting the 4.0 release from
    > > >     Master-10287.7e62d07 as we know it is proven to work based on
    > > company X's
    > > >     actual use of it.
    > > >
    > > >     The truth is (imo), we don't have the bandwidth to manually verify
    > > releases
    > > >     that we are not using/have not used nor do we have the necessary
    > > automated
    > > >     test coverage to verify these releases. So most of our releases are
    > > >     significantly untested.
    > > >
    > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
    > > wrote:
    > > >
    > > >     > I'm really not a fan of allowing Comcast to dictate the release
    > > scope and
    > > >     > schedule. If cherry-picking is too messy, then we can just cut new
    > > minor
    > > >     > releases directly from master.
    > > >     >
    > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com>
    > > wrote:
    > > >     >
    > > >     > > I don't think it's as easy as cherry picking (backporting)
    > > certain
    > > >     > features
    > > >     > > into a release branch. I could be wrong but I really don't think
    > > it is.
    > > >     > >
    > > >     > > So what I'm hearing is that 4.0 gets cut from master and we go
    > > through
    > > >     > with
    > > >     > > our normal process of testing, validating, etc. At the same
    > > time, 4.1
    > > >     > > branch is created from 4.0. Master moves on as normal. Let's say
    > > 25
    > > >     > commits
    > > >     > > come in to master during the next 6 weeks. During that 6 weeks,
    > > we cherry
    > > >     > > pick only the features that look interesting for 4.1? (the rest
    > > just stay
    > > >     > > in master and will get picked up in 5.0?)
    > > >     > >
    > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
    > > weeks apart
    > > >     > and
    > > >     > > have only "interesting" features so the releases are
    > > incrementally very
    > > >     > > small and much easier to test/validate, however, those features
    > > might
    > > >     > > depend on other commits. At some point, cherry picking becomes
    > > impossible
    > > >     > > as the foundation of the code base has shifted so drastically.
    > > Cherry
    > > >     > > picking will become very messy and introduce a high level of
    > > risk imo.
    > > >     > >
    > > >     > > Also, you'd have to manage the scope of these minor releases
    > > which we are
    > > >     > > not very good at because the designated release manager has a
    > > full-time
    > > >     > job
    > > >     > > to attend to and now we have even more releases to
    > > >     > > scope/test/validate/approve. I like the idea of faster releases
    > > but this
    > > >     > > sounds like more work that nobody has the bandwidth for.
    > > >     > >
    > > >     > > Ok, so rather than shooting down an idea with no alternative
    > > proposal, I
    > > >     > > propose this. At Comcast, we are trying to get to the point
    > > where we
    > > >     > > release TC to prod from the head of master every few weeks. What
    > > if every
    > > >     > > time Comcast does an internal upgrade, they note the commit hash
    > > and once
    > > >     > > comcast is happy with the upgrade in their prod enviro, they
    > > propose a
    > > >     > > release pinned to that commit hash? With this approach you'd end
    > > up with:
    > > >     > >
    > > >     > > 1. fast releases (hopefully every few weeks)
    > > >     > > 2. well tested releases as it's in a real prod enviro
    > > >     > > 3. little/no validation work for anyone
    > > >     > >
    > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
    > > front at the
    > > >     > > edge of master and running it in a trusted enviro, they could
    > > propose a
    > > >     > > release as well.
    > > >     > >
    > > >     > > Jeremy
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > >
    > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
    > > david.neuman64@gmail.com>
    > > >     > > wrote:
    > > >     > >
    > > >     > > > I have been thinking about how we can get better with our
    > > release
    > > >     > > cadence.
    > > >     > > > I feel like we have slowed to a crawl and not been as good as
    > > we should
    > > >     > > > about how we release.  Our last Major release was in March and
    > > we
    > > >     > haven't
    > > >     > > > had a real release since.   Moving forward I would like to see
    > > us get
    > > >     > on
    > > >     > > a
    > > >     > > > more consistent cadence and provide smaller releases that can
    > > be more
    > > >     > > > easily digested by the community.  It's my hope that we can
    > > get to a
    > > >     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
    > > >     > releasing
    > > >     > > > in such a way that not all releases all required (e.g. if you
    > > are on
    > > >     > 4.0
    > > >     > > > you don't need 4.1, you can just install 4.2), and we are more
    > > >     > deliberate
    > > >     > > > about what we put into a release.
    > > >     > > >
    > > >     > > > I think we can accomplish this by using our release branches
    > > better.
    > > >     > For
    > > >     > > > each major release we will cut a release branch from master
    > > (e.g. 4.x)
    > > >     > > and
    > > >     > > > then we will use cherry-picking to add new features and bug
    > > fixes to
    > > >     > that
    > > >     > > > release.  This means that if 4.0 has been released and you
    > > want to get
    > > >     > > your
    > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
    > > master and
    > > >     > then
    > > >     > > > cheery pick your squash merged commit to the 4.x branch which
    > > we will
    > > >     > use
    > > >     > > > to create the 4.1 release.  I think we should allow either the
    > > >     > > contributor
    > > >     > > > or the committer (who is merging the PR) to suggest if a
    > > feature goes
    > > >     > > into
    > > >     > > > the release branch, and if we disagree we can take it to the
    > > list.  If
    > > >     > we
    > > >     > > > decide that every PR to master also goes into 4.1 then so be
    > > it, but at
    > > >     > > > least we are making a conscious decision of what goes into the
    > > next
    > > >     > > > release.   This will allow us to not be so worried about what
    > > we are
    > > >     > > > merging into master and how that will affect our next release.
    > > >     > According
    > > >     > > > to our 6 week cadence we will cut a new release from the
    > > current
    > > >     > feature
    > > >     > > > branch and put that up for a vote.  These releases will be
    > > small and
    > > >     > > > testable enough that we can get feedback quickly and move from
    > > RC to
    > > >     > > > release in a short time.  If a release has too many issues
    > > then we may
    > > >     > > end
    > > >     > > > up deciding to skip that minor release in favor of the next
    > > one,
    > > >     > > hopefully
    > > >     > > > this does not happen very often if at all.
    > > >     > > >
    > > >     > > > Once a breaking change is introduced to master which will
    > > require a new
    > > >     > > > major release, we will create a new major release branch from
    > > master
    > > >     > > (which
    > > >     > > > will mean a new release manager).  We will then repeat the
    > > same process
    > > >     > > > with the new release branch, etc, etc.
    > > >     > > >
    > > >     > > > As for LTS we will provide support for the latest major
    > > release plus
    > > >     > the
    > > >     > > > one previous.  So, once we release 4.0 we will support 4.0 and
    > > 3.1.
    > > >     > Any
    > > >     > > > security issues that arise will -- if present -- be applied to
    > > each of
    > > >     > > > these versions.  Once 4.1 is released we will support 4.1 and
    > > 3.1, etc,
    > > >     > > > etc.
    > > >     > > >
    > > >     > > > Please let me know if you have any thoughts on this plan.  If
    > > there are
    > > >     > > no
    > > >     > > > major objections I would like to try this with 4.x with the
    > > idea that
    > > >     > we
    > > >     > > > will adjust how we do things as necessary.
    > > >     > > >
    > > >     > > > We need to get better at releasing and something needs to
    > > change,
    > > >     > > hopefully
    > > >     > > > this can get us going in the right direction.
    > > >     > > >
    > > >     > > > Thanks,
    > > >     > > > Dave
    > > >     > > >
    > > >     > > >
    > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
    > > cherry-picks to
    > > >     > our
    > > >     > > > release branch to control what goes into a minor release.
    > > Major
    > > >     > releases
    > > >     > > > will be cut from master as necessary - such as if there is a
    > > breaking
    > > >     > > > change that is introduced.
    > > >     > > >
    > > >     > >
    > > >     >
    > > >
    > > >
    > >

Re: [EXTERNAL] Re: Release Process

[Rawlin Peters <ra...@apache.org>]

> I'm +1 on this final summary. It infers that we are going to soon be adding a job that runs CI checks on Master. This is good. That's probably a subject for another thread. I do have one question about this approach. I think there is probably a scenario where we need to fix an issue on the 4.x branch itself. In this case I guess we would merge that commit back to master? And then RM would have to keep track of when its safe to fast-forward to the merged version of the fix?

My understanding is that we shouldn't ever have a reason to fix an
issue directly on the 4.x branch itself. When 4.x is ready for a
release, we will cut a new branch (e.g. 4.1.x) off of 4.x at that
point. If we find we absolutely need a fix for 4.1.x, then ideally we
should develop the fix on master then cherry-pick it into 4.1.x. I'm
not sure we'd also cherry-pick it into 4.x since then 4.x would no
longer be a snapshot of the most recent stable commit of master
(meaning we wouldn't be able to just "fast-forward" -- in git
terminology -- to the next most recent stable commit of master).

IMO unless an absolutely necessary fix is also super time-sensitive,
we should develop the fix on master and cherry-pick it into the 4.1.x
branch. Then all cherry-picks would be tracked the same way, without
also having to track hotfixes that need cherry-picked to master.
Whenever cherry-picks/hotfixes are used, the important thing is just
to make sure they get carried forward into the next release (either by
bringing them in naturally via moving forward on master, or
re-cherry-picking them). Again, these should only be things we deem
absolutely necessary for a release (show-stoppers).

- Rawlin

Re: [EXTERNAL] Re: Release Process

["Schmidt, Andrew (Contractor)" <An...@comcast.com>]

I'm +1 on this final summary. It infers that we are going to soon be adding a job that runs CI checks on Master. This is good. That's probably a subject for another thread. I do have one question about this approach. I think there is probably a scenario where we need to fix an issue on the 4.x branch itself. In this case I guess we would merge that commit back to master? And then RM would have to keep track of when its safe to fast-forward to the merged version of the fix?

On 10/31/19, 4:26 PM, "Dave Neuman" <ne...@apache.org> wrote:

    Yeah, I think that about sums it up. Hopefully we can help the release
    manager with some of those responsibilities.

    On Thu, Oct 31, 2019 at 2:31 PM Rawlin Peters <ra...@apache.org> wrote:

    > Alright, I think that sounds fair. In general what I'm hearing is,
    > every 4-6 weeks:
    > 1. fast-forward 4.x to the most recent "stable" commit of master
    > (maybe we automate this and fast-forward every day if some set of
    > tests pass, unless we explicitly know master is unstable for other
    > reasons)
    > 2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
    > 4.*.x that are necessary for the release (e.g. bugs that break the
    > data plane or prevent operators from being able to do their job or run
    > the control plane)
    >
    > Which means the release manager needs to know/track:
    > - unstable commits along with their corresponding fixes
    > - incomplete features
    > - features we'd like to hold until the next release (due to current
    > release having enough already)
    > - any cherry-picks that were deemed necessary for a certain release
    > (so that they can either be carried forward into the next minor
    > release or, ideally, included as part of fast-forwarding 4.x to the
    > most recent stable commit on master)
    >
    > I think this will work. We gain the ability to:
    > 1. let master be "unstable" periodically
    > 2. control the size of releases
    > 3. make releases mirror master as closely as possible (because they
    > will just be a snapshot of the most recent version of master that was
    > considered stable -- unless absolutely necessary bug/security fixes
    > were cherry-picked in)
    >
    > Does that all sound right, Dave?
    >
    > - Rawlin
    >
    >
    >
    >
    >
    >
    >
    > On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com>
    > wrote:
    > >
    > > I think a third reason would be that we are getting very close to our
    > > release branch date, it's a large/complex feature, and we feel like the
    > > release already has enough in it.  In that case we could hold off on
    > adding
    > > that change to the 4.x branch until after we cut the 4.x.x branch.  The
    > > benefit is that we can still merge that feature into master and get all
    > of
    > > our testing (both internally and externally) running against it.   If you
    > > stick with just branching off master and this situation arises then we
    > get
    > > in the spot where we just don't merge the PR at all until after the
    > release
    > > branch is cut (we have two examples of this right now).  I personally
    > think
    > > it's better to get this into master and let our automation run against it
    > > than it is to leave it out in a PR even though it is ready.
    > >
    > > As for the process, I was thinking the same thing except that you don't
    > > even need to cherry-pick.  In your scenario above you can just rebase
    > > master into 4.x once E was merged.  This avoids the headaches of having
    > to
    > > cherry pick each commit while still giving us the benefit of having all
    > of
    > > the changes on the release branch.  We can even decide to setup
    > automation
    > > to rebase master into the release branch on an interval or on demand.  If
    > > we rebase in something that turns out to be broken we just roll back to
    > the
    > > previous rebase (or a certain commit) until that is fixed and then rebase
    > > again.  If a feature gets merged to master that we aren't ready to put
    > into
    > > the release, then we stop the automated rebase until we are ready to get
    > it
    > > into the release -- at that point we should consider ONLY cherry-picking
    > > bug fixes for the remainder of that release cycle to make things easier
    > to
    > > rebase again.
    > >
    > > I am sure there will be unintended consequences, and I am sure we will
    > have
    > > challenges to work through, but I still think that it's worth trying.  We
    > > should fail fast and be willing to change course if necessary.
    > > I think if we can be consistent enough with our releases then maybe we
    > can
    > > discuss branching from master again.
    > >
    > > Thanks,
    > > Dave
    > >
    > >
    > >
    > > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
    > >
    > > > I guess my concern is really about what possible reasons we might have
    > > > to not cherry-pick a specific PR that had been merged into master
    > > > already. The only reasons I can think of to not cherry-pick something
    > > > would be:
    > > > 1. the PR was completely broken (e.g. breaks the build, tests don't
    > > > pass, doesn't compile, etc).
    > > > 2. the PR one one of a larger multi-PR "feature", and *we know*
    > > > releasing some parts early could cause problems (that knowing part is
    > > > crucial)
    > > >
    > > > We should always strive to catch issue #1 in our CI before it merges
    > > > into master. When things like that merge it can block people from
    > > > doing development starting at that point until it's fixed.
    > > > I can see why issue #2 might arise due to multiple different people
    > > > developing one larger feature, but we should still strive to break
    > > > larger features up into individual pieces that can reasonably stand on
    > > > their own if possible. Due to time constraints or other reasons I can
    > > > see that might be a little difficult.
    > > >
    > > > That said, I would still strive to make the 4.x release branch a
    > > > snapshot of master at some "known good point" in time. Let's say we
    > > > cut 4.x at commit A, then a broken commit B gets merged into master.
    > > > If it's completely obvious that it's a broken commit, I'd wait until
    > > > at least a commit C gets merged to fix it. Then I would cherry-pick B
    > > > and C into 4.x at the same time. If the fix is really like a commit E
    > > > with unrelated commits C and D in between, ideally I would still wait
    > > > to cherry-pick C and D until a "fix" commit E was merged (to fix
    > > > commit B), at which point I'd cherrypick commits B through E at the
    > > > same time, in order, to avoid any possible merge conflicts.
    > > >
    > > > I think to make that process easier for myself I would just
    > > > cherry-pick everything into the 4.x branch as it gets merged into
    > > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
    > > > commits were not fixed/completed in time, I would reset the 4.x branch
    > > > to the last stable commit. This would mean that unstable
    > > > commits/incomplete features could potentially block things from going
    > > > into the release. I might count that as a positive because it would
    > > > incentivize us to improve our CI and do our best to also keep the
    > > > master branch as stable as possible.
    > > >
    > > > The end result would be the same as just continually jumping up the
    > > > master branch from known good, stable points in time to the next,
    > > > creating a minor release branch at each of those points.
    > > >
    > > > - Rawlin
    > > >
    > > >
    > > >
    > > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
    > > > >
    > > > > Sorry, I have taken too long to respond.
    > > > >
    > > > > +1000000 on better CI and Testing, I think we all agree that needs
    > to get
    > > > > better and is a different beast than this topic.
    > > > >
    > > > > As for the release branches, I am trying to follow a process similar
    > to
    > > > > ATS, which is usually the model we try to follow with our project.
    > > > > While I agree that branching from master is the ideal solution, we
    > tried
    > > > > that and it didn't work.  I feel like we are trying to agree to do
    > the
    > > > same
    > > > > thing that we already tried and didn't work.  I think we should have
    > a
    > > > > little freedom to commit things into master and, if they break
    > things,
    > > > work
    > > > > on getting them fixed.  We also might commit PRs to master that are
    > > > partial
    > > > > implementations of a bigger feature.   If we are using a time based
    > > > > strategy to cut our releases from master then we are going to be
    > > > > continually worrying about making sure Master stays not broken.  I
    > think
    > > > we
    > > > > should try to keep master deployable, but it's fine if it's not for
    > short
    > > > > periods of time.
    > > > >
    > > > > I am not trying to suggest that we are "picky" about what goes into
    > our
    > > > > release branch.  I think we should try to cherry-pick as much as we
    > can
    > > > > into our release branch once it is ready to be cherry-picked.  I
    > think we
    > > > > should be able to take the tip of our release branch and deploy it
    > > > whenever
    > > > > we want with some confidence since we know that features/bug-fixes
    > that
    > > > > have been cherry-picked have already been reviewed and tested against
    > > > > master.  Yes, its more work, but it also give us the ability to have
    > more
    > > > > confidence in what is in our release at any given time.  Again, we
    > should
    > > > > be trying to cherry-pick in everything that makes sense to
    > cherry-pick.
    > > > >
    > > > > I think the process that I would like to see looks like this:
    > > > > - Cut a 4.x branch from Master
    > > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
    > > > branch
    > > > > - Tag the 4.0.0 release from the 4.0.x branch
    > > > > - New non-breaking changes and bug fixes that go into master are also
    > > > > Cherry Picked to 4.x
    > > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag
    > when we
    > > > > are ready for a RC.
    > > > > - Rinse and repeat for all 4.x releases
    > > > > - Once there's a need for a major release we make a 5.x branch from
    > > > master
    > > > >
    > > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
    > > > then
    > > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
    > > > have
    > > > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's
    > not
    > > > > really that bad and B) it gives us more control.
    > > > >
    > > > > This should allow us to deploy much quicker and with more confidence
    > than
    > > > > is possible today.   We can also choose to deploy from any point in
    > time
    > > > > from the 4.x branch with relatively high confidence.
    > > > >
    > > > > I'll wrap this up by saying that I don't think there's a perfect
    > > > solution,
    > > > > but I think we can agree on something that is worth trying at least
    > with
    > > > > the 4.x release life cycle.  Once we have some experience, we can
    > figure
    > > > > out where it is not meeting our needs and figure out how to adjust as
    > > > > necessary.  I think we can all agree our current process is not
    > working.
    > > > >
    > > > > Thanks,
    > > > > Dave
    > > > >
    > > > >
    > > > >
    > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
    > > > wrote:
    > > > >
    > > > > > +1 on more frequent releases
    > > > > > +1 on improving our CI. We need to get our CI to the point where
    > we're
    > > > > > confident deploying changes to production if the CI passes.
    > > > > > +1 on "trunk-based" development as opposed to GitFlow
    > > > > > -1 on cherry-picking new features. This is risky, tedious, and
    > > > > > represents unnecessary effort that would be better spent elsewhere,
    > > > > > like improving our CI or adding more automated tests.
    > > > > > +1 on cutting minor release branches directly off master
    > > > > >
    > > > > > I think cutting minor release branches directly off master can put
    > us
    > > > > > in a win-win scenario. Our releases become smaller and more
    > frequent,
    > > > > > and we avoid the unnecessary risk and effort of cherry-picking new
    > > > > > features onto stale release branches.
    > > > > >
    > > > > > This is the process that I would propose:
    > > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
    > > > > > 2. Develop master for 4-6 weeks.
    > > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
    > > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
    > > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
    > > > > > cherry-pick the fix from master into the latest release branch
    > (e.g.
    > > > > > 4.1.x), and create a 4.1.1 RC.
    > > > > >
    > > > > > As the 4.x release manager, I really don't want to get into the
    > > > > > business of selecting what should and should not go into a
    > particular
    > > > > > release. Every 4-6 weeks, we should just cut a new minor release
    > > > > > branch off master and take everything that was contributed during
    > that
    > > > > > time, as long as everything builds fine and all the tests pass. The
    > > > > > master branch, while it doesn't always have to be perfect, should
    > at
    > > > > > least always be deployable. If a new feature is merged with bugs
    > and
    > > > > > becomes a show-stopper, it should be either immediately remedied or
    > > > > > reverted on master while we work on a fix.
    > > > > >
    > > > > > - Rawlin
    > > > > >
    > > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
    > > > > > <Mi...@comcast.com> wrote:
    > > > > > >
    > > > > > > In my opinion, in order to get to a cadence we are discussing we
    > > > need to
    > > > > > put a lot more work into the CI system. It has been failing
    > > > consistently,
    > > > > > doesn’t block PR approvals/merges and when it actually runs it
    > does not
    > > > > > test anything besides build and license headers.
    > > > > > >
    > > > > > > In the past couple of weeks we have had (off the top of my head):
    > > > > > >
    > > > > > > * Unit tests broken
    > > > > > > * TO API tests broken
    > > > > > > * Golang vendor issues
    > > > > > > * TO Go build issues
    > > > > > >
    > > > > > > All were merged and broke master.
    > > > > > >
    > > > > > > Yes, having release branches sounds great and more aggressive
    > cadence
    > > > > > will minimize the amount of risk but I think that comes with the
    > need
    > > > to
    > > > > > improve on our automated validation and testing.
    > > > > > >
    > > > > > > We have proven that we have not kept master in a good state and
    > > > adding
    > > > > > more releases (more branches) will make it even harder to keep that
    > > > > > stability we are already fighting.
    > > > > > >
    > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
    > > > wrote:
    > > > > > >
    > > > > > >     Yeah, I get it. No one company should be driving release
    > > > > > schedules/scope.
    > > > > > >
    > > > > > >     What I was really getting at is if ANY company has a recent
    > > > version
    > > > > > in a
    > > > > > >     test or prod-like enviro (meaning the version is being
    > exercised
    > > > and
    > > > > > >     thoroughly tested), then maybe we consider simply cutting a
    > > > release
    > > > > > from
    > > > > > >     that version. For example, imagine company X had this
    > version in
    > > > a
    > > > > > >     test/prod enviro:
    > > > > > >
    > > > > > >     Master-10287.7e62d07 (
    > > > > > >
    > > > > >
    > > >
    > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
    > > > > > )
    > > > > > >
    > > > > > >     Then I would be in favor of cutting the 4.0 release from
    > > > > > >     Master-10287.7e62d07 as we know it is proven to work based on
    > > > > > company X's
    > > > > > >     actual use of it.
    > > > > > >
    > > > > > >     The truth is (imo), we don't have the bandwidth to manually
    > > > verify
    > > > > > releases
    > > > > > >     that we are not using/have not used nor do we have the
    > necessary
    > > > > > automated
    > > > > > >     test coverage to verify these releases. So most of our
    > releases
    > > > are
    > > > > > >     significantly untested.
    > > > > > >
    > > > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
    > ocket8888@gmail.com>
    > > > > > wrote:
    > > > > > >
    > > > > > >     > I'm really not a fan of allowing Comcast to dictate the
    > release
    > > > > > scope and
    > > > > > >     > schedule. If cherry-picking is too messy, then we can just
    > cut
    > > > new
    > > > > > minor
    > > > > > >     > releases directly from master.
    > > > > > >     >
    > > > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
    > > > mitchell852@gmail.com>
    > > > > > wrote:
    > > > > > >     >
    > > > > > >     > > I don't think it's as easy as cherry picking
    > (backporting)
    > > > > > certain
    > > > > > >     > features
    > > > > > >     > > into a release branch. I could be wrong but I really
    > don't
    > > > think
    > > > > > it is.
    > > > > > >     > >
    > > > > > >     > > So what I'm hearing is that 4.0 gets cut from master and
    > we
    > > > go
    > > > > > through
    > > > > > >     > with
    > > > > > >     > > our normal process of testing, validating, etc. At the
    > same
    > > > > > time, 4.1
    > > > > > >     > > branch is created from 4.0. Master moves on as normal.
    > Let's
    > > > say
    > > > > > 25
    > > > > > >     > commits
    > > > > > >     > > come in to master during the next 6 weeks. During that 6
    > > > weeks,
    > > > > > we cherry
    > > > > > >     > > pick only the features that look interesting for 4.1?
    > (the
    > > > rest
    > > > > > just stay
    > > > > > >     > > in master and will get picked up in 5.0?)
    > > > > > >     > >
    > > > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
    > roughly 6
    > > > > > weeks apart
    > > > > > >     > and
    > > > > > >     > > have only "interesting" features so the releases are
    > > > > > incrementally very
    > > > > > >     > > small and much easier to test/validate, however, those
    > > > features
    > > > > > might
    > > > > > >     > > depend on other commits. At some point, cherry picking
    > > > becomes
    > > > > > impossible
    > > > > > >     > > as the foundation of the code base has shifted so
    > > > drastically.
    > > > > > Cherry
    > > > > > >     > > picking will become very messy and introduce a high
    > level of
    > > > > > risk imo.
    > > > > > >     > >
    > > > > > >     > > Also, you'd have to manage the scope of these minor
    > releases
    > > > > > which we are
    > > > > > >     > > not very good at because the designated release manager
    > has a
    > > > > > full-time
    > > > > > >     > job
    > > > > > >     > > to attend to and now we have even more releases to
    > > > > > >     > > scope/test/validate/approve. I like the idea of faster
    > > > releases
    > > > > > but this
    > > > > > >     > > sounds like more work that nobody has the bandwidth for.
    > > > > > >     > >
    > > > > > >     > > Ok, so rather than shooting down an idea with no
    > alternative
    > > > > > proposal, I
    > > > > > >     > > propose this. At Comcast, we are trying to get to the
    > point
    > > > > > where we
    > > > > > >     > > release TC to prod from the head of master every few
    > weeks.
    > > > What
    > > > > > if every
    > > > > > >     > > time Comcast does an internal upgrade, they note the
    > commit
    > > > hash
    > > > > > and once
    > > > > > >     > > comcast is happy with the upgrade in their prod enviro,
    > they
    > > > > > propose a
    > > > > > >     > > release pinned to that commit hash? With this approach
    > you'd
    > > > end
    > > > > > up with:
    > > > > > >     > >
    > > > > > >     > > 1. fast releases (hopefully every few weeks)
    > > > > > >     > > 2. well tested releases as it's in a real prod enviro
    > > > > > >     > > 3. little/no validation work for anyone
    > > > > > >     > >
    > > > > > >     > > Also, this doesn't have to be Comcast. If anyone else is
    > out
    > > > > > front at the
    > > > > > >     > > edge of master and running it in a trusted enviro, they
    > could
    > > > > > propose a
    > > > > > >     > > release as well.
    > > > > > >     > >
    > > > > > >     > > Jeremy
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > >
    > > > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
    > > > > > david.neuman64@gmail.com>
    > > > > > >     > > wrote:
    > > > > > >     > >
    > > > > > >     > > > I have been thinking about how we can get better with
    > our
    > > > > > release
    > > > > > >     > > cadence.
    > > > > > >     > > > I feel like we have slowed to a crawl and not been as
    > good
    > > > as
    > > > > > we should
    > > > > > >     > > > about how we release.  Our last Major release was in
    > March
    > > > and
    > > > > > we
    > > > > > >     > haven't
    > > > > > >     > > > had a real release since.   Moving forward I would
    > like to
    > > > see
    > > > > > us get
    > > > > > >     > on
    > > > > > >     > > a
    > > > > > >     > > > more consistent cadence and provide smaller releases
    > that
    > > > can
    > > > > > be more
    > > > > > >     > > > easily digested by the community.  It's my hope that
    > we can
    > > > > > get to a
    > > > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
    > > > we are
    > > > > > >     > releasing
    > > > > > >     > > > in such a way that not all releases all required (e.g.
    > if
    > > > you
    > > > > > are on
    > > > > > >     > 4.0
    > > > > > >     > > > you don't need 4.1, you can just install 4.2), and we
    > are
    > > > more
    > > > > > >     > deliberate
    > > > > > >     > > > about what we put into a release.
    > > > > > >     > > >
    > > > > > >     > > > I think we can accomplish this by using our release
    > > > branches
    > > > > > better.
    > > > > > >     > For
    > > > > > >     > > > each major release we will cut a release branch from
    > master
    > > > > > (e.g. 4.x)
    > > > > > >     > > and
    > > > > > >     > > > then we will use cherry-picking to add new features
    > and bug
    > > > > > fixes to
    > > > > > >     > that
    > > > > > >     > > > release.  This means that if 4.0 has been released and
    > you
    > > > > > want to get
    > > > > > >     > > your
    > > > > > >     > > > feature/bug fix in 4.1, you will first submit your PR
    > to
    > > > > > master and
    > > > > > >     > then
    > > > > > >     > > > cheery pick your squash merged commit to the 4.x branch
    > > > which
    > > > > > we will
    > > > > > >     > use
    > > > > > >     > > > to create the 4.1 release.  I think we should allow
    > either
    > > > the
    > > > > > >     > > contributor
    > > > > > >     > > > or the committer (who is merging the PR) to suggest if
    > a
    > > > > > feature goes
    > > > > > >     > > into
    > > > > > >     > > > the release branch, and if we disagree we can take it
    > to
    > > > the
    > > > > > list.  If
    > > > > > >     > we
    > > > > > >     > > > decide that every PR to master also goes into 4.1 then
    > so
    > > > be
    > > > > > it, but at
    > > > > > >     > > > least we are making a conscious decision of what goes
    > into
    > > > the
    > > > > > next
    > > > > > >     > > > release.   This will allow us to not be so worried
    > about
    > > > what
    > > > > > we are
    > > > > > >     > > > merging into master and how that will affect our next
    > > > release.
    > > > > > >     > According
    > > > > > >     > > > to our 6 week cadence we will cut a new release from
    > the
    > > > > > current
    > > > > > >     > feature
    > > > > > >     > > > branch and put that up for a vote.  These releases
    > will be
    > > > > > small and
    > > > > > >     > > > testable enough that we can get feedback quickly and
    > move
    > > > from
    > > > > > RC to
    > > > > > >     > > > release in a short time.  If a release has too many
    > issues
    > > > > > then we may
    > > > > > >     > > end
    > > > > > >     > > > up deciding to skip that minor release in favor of the
    > next
    > > > > > one,
    > > > > > >     > > hopefully
    > > > > > >     > > > this does not happen very often if at all.
    > > > > > >     > > >
    > > > > > >     > > > Once a breaking change is introduced to master which
    > will
    > > > > > require a new
    > > > > > >     > > > major release, we will create a new major release
    > branch
    > > > from
    > > > > > master
    > > > > > >     > > (which
    > > > > > >     > > > will mean a new release manager).  We will then repeat
    > the
    > > > > > same process
    > > > > > >     > > > with the new release branch, etc, etc.
    > > > > > >     > > >
    > > > > > >     > > > As for LTS we will provide support for the latest major
    > > > > > release plus
    > > > > > >     > the
    > > > > > >     > > > one previous.  So, once we release 4.0 we will support
    > 4.0
    > > > and
    > > > > > 3.1.
    > > > > > >     > Any
    > > > > > >     > > > security issues that arise will -- if present -- be
    > > > applied to
    > > > > > each of
    > > > > > >     > > > these versions.  Once 4.1 is released we will support
    > 4.1
    > > > and
    > > > > > 3.1, etc,
    > > > > > >     > > > etc.
    > > > > > >     > > >
    > > > > > >     > > > Please let me know if you have any thoughts on this
    > plan.
    > > > If
    > > > > > there are
    > > > > > >     > > no
    > > > > > >     > > > major objections I would like to try this with 4.x
    > with the
    > > > > > idea that
    > > > > > >     > we
    > > > > > >     > > > will adjust how we do things as necessary.
    > > > > > >     > > >
    > > > > > >     > > > We need to get better at releasing and something needs
    > to
    > > > > > change,
    > > > > > >     > > hopefully
    > > > > > >     > > > this can get us going in the right direction.
    > > > > > >     > > >
    > > > > > >     > > > Thanks,
    > > > > > >     > > > Dave
    > > > > > >     > > >
    > > > > > >     > > >
    > > > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
    > > > > > cherry-picks to
    > > > > > >     > our
    > > > > > >     > > > release branch to control what goes into a minor
    > release.
    > > > > > Major
    > > > > > >     > releases
    > > > > > >     > > > will be cut from master as necessary - such as if
    > there is
    > > > a
    > > > > > breaking
    > > > > > >     > > > change that is introduced.
    > > > > > >     > > >
    > > > > > >     > >
    > > > > > >     >
    > > > > > >
    > > > > > >
    > > > > >
    > > >
    >

Re: [EXTERNAL] Re: Release Process

[Dave Neuman <ne...@apache.org>]

Yeah, I think that about sums it up. Hopefully we can help the release
manager with some of those responsibilities.

On Thu, Oct 31, 2019 at 2:31 PM Rawlin Peters <ra...@apache.org> wrote:

> Alright, I think that sounds fair. In general what I'm hearing is,
> every 4-6 weeks:
> 1. fast-forward 4.x to the most recent "stable" commit of master
> (maybe we automate this and fast-forward every day if some set of
> tests pass, unless we explicitly know master is unstable for other
> reasons)
> 2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
> 4.*.x that are necessary for the release (e.g. bugs that break the
> data plane or prevent operators from being able to do their job or run
> the control plane)
>
> Which means the release manager needs to know/track:
> - unstable commits along with their corresponding fixes
> - incomplete features
> - features we'd like to hold until the next release (due to current
> release having enough already)
> - any cherry-picks that were deemed necessary for a certain release
> (so that they can either be carried forward into the next minor
> release or, ideally, included as part of fast-forwarding 4.x to the
> most recent stable commit on master)
>
> I think this will work. We gain the ability to:
> 1. let master be "unstable" periodically
> 2. control the size of releases
> 3. make releases mirror master as closely as possible (because they
> will just be a snapshot of the most recent version of master that was
> considered stable -- unless absolutely necessary bug/security fixes
> were cherry-picked in)
>
> Does that all sound right, Dave?
>
> - Rawlin
>
>
>
>
>
>
>
> On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com>
> wrote:
> >
> > I think a third reason would be that we are getting very close to our
> > release branch date, it's a large/complex feature, and we feel like the
> > release already has enough in it.  In that case we could hold off on
> adding
> > that change to the 4.x branch until after we cut the 4.x.x branch.  The
> > benefit is that we can still merge that feature into master and get all
> of
> > our testing (both internally and externally) running against it.   If you
> > stick with just branching off master and this situation arises then we
> get
> > in the spot where we just don't merge the PR at all until after the
> release
> > branch is cut (we have two examples of this right now).  I personally
> think
> > it's better to get this into master and let our automation run against it
> > than it is to leave it out in a PR even though it is ready.
> >
> > As for the process, I was thinking the same thing except that you don't
> > even need to cherry-pick.  In your scenario above you can just rebase
> > master into 4.x once E was merged.  This avoids the headaches of having
> to
> > cherry pick each commit while still giving us the benefit of having all
> of
> > the changes on the release branch.  We can even decide to setup
> automation
> > to rebase master into the release branch on an interval or on demand.  If
> > we rebase in something that turns out to be broken we just roll back to
> the
> > previous rebase (or a certain commit) until that is fixed and then rebase
> > again.  If a feature gets merged to master that we aren't ready to put
> into
> > the release, then we stop the automated rebase until we are ready to get
> it
> > into the release -- at that point we should consider ONLY cherry-picking
> > bug fixes for the remainder of that release cycle to make things easier
> to
> > rebase again.
> >
> > I am sure there will be unintended consequences, and I am sure we will
> have
> > challenges to work through, but I still think that it's worth trying.  We
> > should fail fast and be willing to change course if necessary.
> > I think if we can be consistent enough with our releases then maybe we
> can
> > discuss branching from master again.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > I guess my concern is really about what possible reasons we might have
> > > to not cherry-pick a specific PR that had been merged into master
> > > already. The only reasons I can think of to not cherry-pick something
> > > would be:
> > > 1. the PR was completely broken (e.g. breaks the build, tests don't
> > > pass, doesn't compile, etc).
> > > 2. the PR one one of a larger multi-PR "feature", and *we know*
> > > releasing some parts early could cause problems (that knowing part is
> > > crucial)
> > >
> > > We should always strive to catch issue #1 in our CI before it merges
> > > into master. When things like that merge it can block people from
> > > doing development starting at that point until it's fixed.
> > > I can see why issue #2 might arise due to multiple different people
> > > developing one larger feature, but we should still strive to break
> > > larger features up into individual pieces that can reasonably stand on
> > > their own if possible. Due to time constraints or other reasons I can
> > > see that might be a little difficult.
> > >
> > > That said, I would still strive to make the 4.x release branch a
> > > snapshot of master at some "known good point" in time. Let's say we
> > > cut 4.x at commit A, then a broken commit B gets merged into master.
> > > If it's completely obvious that it's a broken commit, I'd wait until
> > > at least a commit C gets merged to fix it. Then I would cherry-pick B
> > > and C into 4.x at the same time. If the fix is really like a commit E
> > > with unrelated commits C and D in between, ideally I would still wait
> > > to cherry-pick C and D until a "fix" commit E was merged (to fix
> > > commit B), at which point I'd cherrypick commits B through E at the
> > > same time, in order, to avoid any possible merge conflicts.
> > >
> > > I think to make that process easier for myself I would just
> > > cherry-pick everything into the 4.x branch as it gets merged into
> > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
> > > commits were not fixed/completed in time, I would reset the 4.x branch
> > > to the last stable commit. This would mean that unstable
> > > commits/incomplete features could potentially block things from going
> > > into the release. I might count that as a positive because it would
> > > incentivize us to improve our CI and do our best to also keep the
> > > master branch as stable as possible.
> > >
> > > The end result would be the same as just continually jumping up the
> > > master branch from known good, stable points in time to the next,
> > > creating a minor release branch at each of those points.
> > >
> > > - Rawlin
> > >
> > >
> > >
> > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
> > > >
> > > > Sorry, I have taken too long to respond.
> > > >
> > > > +1000000 on better CI and Testing, I think we all agree that needs
> to get
> > > > better and is a different beast than this topic.
> > > >
> > > > As for the release branches, I am trying to follow a process similar
> to
> > > > ATS, which is usually the model we try to follow with our project.
> > > > While I agree that branching from master is the ideal solution, we
> tried
> > > > that and it didn't work.  I feel like we are trying to agree to do
> the
> > > same
> > > > thing that we already tried and didn't work.  I think we should have
> a
> > > > little freedom to commit things into master and, if they break
> things,
> > > work
> > > > on getting them fixed.  We also might commit PRs to master that are
> > > partial
> > > > implementations of a bigger feature.   If we are using a time based
> > > > strategy to cut our releases from master then we are going to be
> > > > continually worrying about making sure Master stays not broken.  I
> think
> > > we
> > > > should try to keep master deployable, but it's fine if it's not for
> short
> > > > periods of time.
> > > >
> > > > I am not trying to suggest that we are "picky" about what goes into
> our
> > > > release branch.  I think we should try to cherry-pick as much as we
> can
> > > > into our release branch once it is ready to be cherry-picked.  I
> think we
> > > > should be able to take the tip of our release branch and deploy it
> > > whenever
> > > > we want with some confidence since we know that features/bug-fixes
> that
> > > > have been cherry-picked have already been reviewed and tested against
> > > > master.  Yes, its more work, but it also give us the ability to have
> more
> > > > confidence in what is in our release at any given time.  Again, we
> should
> > > > be trying to cherry-pick in everything that makes sense to
> cherry-pick.
> > > >
> > > > I think the process that I would like to see looks like this:
> > > > - Cut a 4.x branch from Master
> > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > > branch
> > > > - Tag the 4.0.0 release from the 4.0.x branch
> > > > - New non-breaking changes and bug fixes that go into master are also
> > > > Cherry Picked to 4.x
> > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag
> when we
> > > > are ready for a RC.
> > > > - Rinse and repeat for all 4.x releases
> > > > - Once there's a need for a major release we make a 5.x branch from
> > > master
> > > >
> > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > > then
> > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > > have
> > > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's
> not
> > > > really that bad and B) it gives us more control.
> > > >
> > > > This should allow us to deploy much quicker and with more confidence
> than
> > > > is possible today.   We can also choose to deploy from any point in
> time
> > > > from the 4.x branch with relatively high confidence.
> > > >
> > > > I'll wrap this up by saying that I don't think there's a perfect
> > > solution,
> > > > but I think we can agree on something that is worth trying at least
> with
> > > > the 4.x release life cycle.  Once we have some experience, we can
> figure
> > > > out where it is not meeting our needs and figure out how to adjust as
> > > > necessary.  I think we can all agree our current process is not
> working.
> > > >
> > > > Thanks,
> > > > Dave
> > > >
> > > >
> > > >
> > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > > wrote:
> > > >
> > > > > +1 on more frequent releases
> > > > > +1 on improving our CI. We need to get our CI to the point where
> we're
> > > > > confident deploying changes to production if the CI passes.
> > > > > +1 on "trunk-based" development as opposed to GitFlow
> > > > > -1 on cherry-picking new features. This is risky, tedious, and
> > > > > represents unnecessary effort that would be better spent elsewhere,
> > > > > like improving our CI or adding more automated tests.
> > > > > +1 on cutting minor release branches directly off master
> > > > >
> > > > > I think cutting minor release branches directly off master can put
> us
> > > > > in a win-win scenario. Our releases become smaller and more
> frequent,
> > > > > and we avoid the unnecessary risk and effort of cherry-picking new
> > > > > features onto stale release branches.
> > > > >
> > > > > This is the process that I would propose:
> > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > > > > 2. Develop master for 4-6 weeks.
> > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
> > > > > cherry-pick the fix from master into the latest release branch
> (e.g.
> > > > > 4.1.x), and create a 4.1.1 RC.
> > > > >
> > > > > As the 4.x release manager, I really don't want to get into the
> > > > > business of selecting what should and should not go into a
> particular
> > > > > release. Every 4-6 weeks, we should just cut a new minor release
> > > > > branch off master and take everything that was contributed during
> that
> > > > > time, as long as everything builds fine and all the tests pass. The
> > > > > master branch, while it doesn't always have to be perfect, should
> at
> > > > > least always be deployable. If a new feature is merged with bugs
> and
> > > > > becomes a show-stopper, it should be either immediately remedied or
> > > > > reverted on master while we work on a fix.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > > > > <Mi...@comcast.com> wrote:
> > > > > >
> > > > > > In my opinion, in order to get to a cadence we are discussing we
> > > need to
> > > > > put a lot more work into the CI system. It has been failing
> > > consistently,
> > > > > doesn’t block PR approvals/merges and when it actually runs it
> does not
> > > > > test anything besides build and license headers.
> > > > > >
> > > > > > In the past couple of weeks we have had (off the top of my head):
> > > > > >
> > > > > > * Unit tests broken
> > > > > > * TO API tests broken
> > > > > > * Golang vendor issues
> > > > > > * TO Go build issues
> > > > > >
> > > > > > All were merged and broke master.
> > > > > >
> > > > > > Yes, having release branches sounds great and more aggressive
> cadence
> > > > > will minimize the amount of risk but I think that comes with the
> need
> > > to
> > > > > improve on our automated validation and testing.
> > > > > >
> > > > > > We have proven that we have not kept master in a good state and
> > > adding
> > > > > more releases (more branches) will make it even harder to keep that
> > > > > stability we are already fighting.
> > > > > >
> > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >     Yeah, I get it. No one company should be driving release
> > > > > schedules/scope.
> > > > > >
> > > > > >     What I was really getting at is if ANY company has a recent
> > > version
> > > > > in a
> > > > > >     test or prod-like enviro (meaning the version is being
> exercised
> > > and
> > > > > >     thoroughly tested), then maybe we consider simply cutting a
> > > release
> > > > > from
> > > > > >     that version. For example, imagine company X had this
> version in
> > > a
> > > > > >     test/prod enviro:
> > > > > >
> > > > > >     Master-10287.7e62d07 (
> > > > > >
> > > > >
> > >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > > > > )
> > > > > >
> > > > > >     Then I would be in favor of cutting the 4.0 release from
> > > > > >     Master-10287.7e62d07 as we know it is proven to work based on
> > > > > company X's
> > > > > >     actual use of it.
> > > > > >
> > > > > >     The truth is (imo), we don't have the bandwidth to manually
> > > verify
> > > > > releases
> > > > > >     that we are not using/have not used nor do we have the
> necessary
> > > > > automated
> > > > > >     test coverage to verify these releases. So most of our
> releases
> > > are
> > > > > >     significantly untested.
> > > > > >
> > > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> ocket8888@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >     > I'm really not a fan of allowing Comcast to dictate the
> release
> > > > > scope and
> > > > > >     > schedule. If cherry-picking is too messy, then we can just
> cut
> > > new
> > > > > minor
> > > > > >     > releases directly from master.
> > > > > >     >
> > > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > > mitchell852@gmail.com>
> > > > > wrote:
> > > > > >     >
> > > > > >     > > I don't think it's as easy as cherry picking
> (backporting)
> > > > > certain
> > > > > >     > features
> > > > > >     > > into a release branch. I could be wrong but I really
> don't
> > > think
> > > > > it is.
> > > > > >     > >
> > > > > >     > > So what I'm hearing is that 4.0 gets cut from master and
> we
> > > go
> > > > > through
> > > > > >     > with
> > > > > >     > > our normal process of testing, validating, etc. At the
> same
> > > > > time, 4.1
> > > > > >     > > branch is created from 4.0. Master moves on as normal.
> Let's
> > > say
> > > > > 25
> > > > > >     > commits
> > > > > >     > > come in to master during the next 6 weeks. During that 6
> > > weeks,
> > > > > we cherry
> > > > > >     > > pick only the features that look interesting for 4.1?
> (the
> > > rest
> > > > > just stay
> > > > > >     > > in master and will get picked up in 5.0?)
> > > > > >     > >
> > > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> roughly 6
> > > > > weeks apart
> > > > > >     > and
> > > > > >     > > have only "interesting" features so the releases are
> > > > > incrementally very
> > > > > >     > > small and much easier to test/validate, however, those
> > > features
> > > > > might
> > > > > >     > > depend on other commits. At some point, cherry picking
> > > becomes
> > > > > impossible
> > > > > >     > > as the foundation of the code base has shifted so
> > > drastically.
> > > > > Cherry
> > > > > >     > > picking will become very messy and introduce a high
> level of
> > > > > risk imo.
> > > > > >     > >
> > > > > >     > > Also, you'd have to manage the scope of these minor
> releases
> > > > > which we are
> > > > > >     > > not very good at because the designated release manager
> has a
> > > > > full-time
> > > > > >     > job
> > > > > >     > > to attend to and now we have even more releases to
> > > > > >     > > scope/test/validate/approve. I like the idea of faster
> > > releases
> > > > > but this
> > > > > >     > > sounds like more work that nobody has the bandwidth for.
> > > > > >     > >
> > > > > >     > > Ok, so rather than shooting down an idea with no
> alternative
> > > > > proposal, I
> > > > > >     > > propose this. At Comcast, we are trying to get to the
> point
> > > > > where we
> > > > > >     > > release TC to prod from the head of master every few
> weeks.
> > > What
> > > > > if every
> > > > > >     > > time Comcast does an internal upgrade, they note the
> commit
> > > hash
> > > > > and once
> > > > > >     > > comcast is happy with the upgrade in their prod enviro,
> they
> > > > > propose a
> > > > > >     > > release pinned to that commit hash? With this approach
> you'd
> > > end
> > > > > up with:
> > > > > >     > >
> > > > > >     > > 1. fast releases (hopefully every few weeks)
> > > > > >     > > 2. well tested releases as it's in a real prod enviro
> > > > > >     > > 3. little/no validation work for anyone
> > > > > >     > >
> > > > > >     > > Also, this doesn't have to be Comcast. If anyone else is
> out
> > > > > front at the
> > > > > >     > > edge of master and running it in a trusted enviro, they
> could
> > > > > propose a
> > > > > >     > > release as well.
> > > > > >     > >
> > > > > >     > > Jeremy
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > >
> > > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > > > > david.neuman64@gmail.com>
> > > > > >     > > wrote:
> > > > > >     > >
> > > > > >     > > > I have been thinking about how we can get better with
> our
> > > > > release
> > > > > >     > > cadence.
> > > > > >     > > > I feel like we have slowed to a crawl and not been as
> good
> > > as
> > > > > we should
> > > > > >     > > > about how we release.  Our last Major release was in
> March
> > > and
> > > > > we
> > > > > >     > haven't
> > > > > >     > > > had a real release since.   Moving forward I would
> like to
> > > see
> > > > > us get
> > > > > >     > on
> > > > > >     > > a
> > > > > >     > > > more consistent cadence and provide smaller releases
> that
> > > can
> > > > > be more
> > > > > >     > > > easily digested by the community.  It's my hope that
> we can
> > > > > get to a
> > > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> > > we are
> > > > > >     > releasing
> > > > > >     > > > in such a way that not all releases all required (e.g.
> if
> > > you
> > > > > are on
> > > > > >     > 4.0
> > > > > >     > > > you don't need 4.1, you can just install 4.2), and we
> are
> > > more
> > > > > >     > deliberate
> > > > > >     > > > about what we put into a release.
> > > > > >     > > >
> > > > > >     > > > I think we can accomplish this by using our release
> > > branches
> > > > > better.
> > > > > >     > For
> > > > > >     > > > each major release we will cut a release branch from
> master
> > > > > (e.g. 4.x)
> > > > > >     > > and
> > > > > >     > > > then we will use cherry-picking to add new features
> and bug
> > > > > fixes to
> > > > > >     > that
> > > > > >     > > > release.  This means that if 4.0 has been released and
> you
> > > > > want to get
> > > > > >     > > your
> > > > > >     > > > feature/bug fix in 4.1, you will first submit your PR
> to
> > > > > master and
> > > > > >     > then
> > > > > >     > > > cheery pick your squash merged commit to the 4.x branch
> > > which
> > > > > we will
> > > > > >     > use
> > > > > >     > > > to create the 4.1 release.  I think we should allow
> either
> > > the
> > > > > >     > > contributor
> > > > > >     > > > or the committer (who is merging the PR) to suggest if
> a
> > > > > feature goes
> > > > > >     > > into
> > > > > >     > > > the release branch, and if we disagree we can take it
> to
> > > the
> > > > > list.  If
> > > > > >     > we
> > > > > >     > > > decide that every PR to master also goes into 4.1 then
> so
> > > be
> > > > > it, but at
> > > > > >     > > > least we are making a conscious decision of what goes
> into
> > > the
> > > > > next
> > > > > >     > > > release.   This will allow us to not be so worried
> about
> > > what
> > > > > we are
> > > > > >     > > > merging into master and how that will affect our next
> > > release.
> > > > > >     > According
> > > > > >     > > > to our 6 week cadence we will cut a new release from
> the
> > > > > current
> > > > > >     > feature
> > > > > >     > > > branch and put that up for a vote.  These releases
> will be
> > > > > small and
> > > > > >     > > > testable enough that we can get feedback quickly and
> move
> > > from
> > > > > RC to
> > > > > >     > > > release in a short time.  If a release has too many
> issues
> > > > > then we may
> > > > > >     > > end
> > > > > >     > > > up deciding to skip that minor release in favor of the
> next
> > > > > one,
> > > > > >     > > hopefully
> > > > > >     > > > this does not happen very often if at all.
> > > > > >     > > >
> > > > > >     > > > Once a breaking change is introduced to master which
> will
> > > > > require a new
> > > > > >     > > > major release, we will create a new major release
> branch
> > > from
> > > > > master
> > > > > >     > > (which
> > > > > >     > > > will mean a new release manager).  We will then repeat
> the
> > > > > same process
> > > > > >     > > > with the new release branch, etc, etc.
> > > > > >     > > >
> > > > > >     > > > As for LTS we will provide support for the latest major
> > > > > release plus
> > > > > >     > the
> > > > > >     > > > one previous.  So, once we release 4.0 we will support
> 4.0
> > > and
> > > > > 3.1.
> > > > > >     > Any
> > > > > >     > > > security issues that arise will -- if present -- be
> > > applied to
> > > > > each of
> > > > > >     > > > these versions.  Once 4.1 is released we will support
> 4.1
> > > and
> > > > > 3.1, etc,
> > > > > >     > > > etc.
> > > > > >     > > >
> > > > > >     > > > Please let me know if you have any thoughts on this
> plan.
> > > If
> > > > > there are
> > > > > >     > > no
> > > > > >     > > > major objections I would like to try this with 4.x
> with the
> > > > > idea that
> > > > > >     > we
> > > > > >     > > > will adjust how we do things as necessary.
> > > > > >     > > >
> > > > > >     > > > We need to get better at releasing and something needs
> to
> > > > > change,
> > > > > >     > > hopefully
> > > > > >     > > > this can get us going in the right direction.
> > > > > >     > > >
> > > > > >     > > > Thanks,
> > > > > >     > > > Dave
> > > > > >     > > >
> > > > > >     > > >
> > > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > > > > cherry-picks to
> > > > > >     > our
> > > > > >     > > > release branch to control what goes into a minor
> release.
> > > > > Major
> > > > > >     > releases
> > > > > >     > > > will be cut from master as necessary - such as if
> there is
> > > a
> > > > > breaking
> > > > > >     > > > change that is introduced.
> > > > > >     > > >
> > > > > >     > >
> > > > > >     >
> > > > > >
> > > > > >
> > > > >
> > >
>

Re: [EXTERNAL] Re: Release Process

["Genz, Geoffrey" <Ge...@comcast.com>]

We are in violent agreement and that was sort of the point I was making.  I’m wholly in favor of release process that encourages development that tracks master as close as possible and discourages long lived branches.  Features imho should be written in a way that is always seamlessly (as much as possible) integrated into the existing code base and using feature flags is one way to accomplish that without risking (too much) the stability of that code base.

Geoff

On 10/31/19, 3:19 PM, "Rawlin Peters" <ra...@apache.org> wrote:

    I agree, but this release process wouldn't lead to long-lived feature
    branches since all development would still be done on master and
    merged into master. Unstable or incomplete features could potentially
    delay releases from going out, but I agree that feature flags could
    mitigate those problems. Anything we can do to keep the master branch
    deployable is a +1 in my book.

    - Rawlin

    On Thu, Oct 31, 2019 at 2:46 PM Genz, Geoffrey
    <Ge...@comcast.com> wrote:
    >
    > Just throwing out that the judicious use of feature flags could reduce the pain from unstable or incomplete new features.  Long lived feature branches are something I believe should be avoided.
    >
    > Geoff
    >
    > On 10/31/19, 2:32 PM, "Rawlin Peters" <ra...@apache.org> wrote:
    >
    >     Alright, I think that sounds fair. In general what I'm hearing is,
    >     every 4-6 weeks:
    >     1. fast-forward 4.x to the most recent "stable" commit of master
    >     (maybe we automate this and fast-forward every day if some set of
    >     tests pass, unless we explicitly know master is unstable for other
    >     reasons)
    >     2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
    >     4.*.x that are necessary for the release (e.g. bugs that break the
    >     data plane or prevent operators from being able to do their job or run
    >     the control plane)
    >
    >     Which means the release manager needs to know/track:
    >     - unstable commits along with their corresponding fixes
    >     - incomplete features
    >     - features we'd like to hold until the next release (due to current
    >     release having enough already)
    >     - any cherry-picks that were deemed necessary for a certain release
    >     (so that they can either be carried forward into the next minor
    >     release or, ideally, included as part of fast-forwarding 4.x to the
    >     most recent stable commit on master)
    >
    >     I think this will work. We gain the ability to:
    >     1. let master be "unstable" periodically
    >     2. control the size of releases
    >     3. make releases mirror master as closely as possible (because they
    >     will just be a snapshot of the most recent version of master that was
    >     considered stable -- unless absolutely necessary bug/security fixes
    >     were cherry-picked in)
    >
    >     Does that all sound right, Dave?
    >
    >     - Rawlin
    >
    >
    >
    >
    >
    >
    >
    >     On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com> wrote:
    >     >
    >     > I think a third reason would be that we are getting very close to our
    >     > release branch date, it's a large/complex feature, and we feel like the
    >     > release already has enough in it.  In that case we could hold off on adding
    >     > that change to the 4.x branch until after we cut the 4.x.x branch.  The
    >     > benefit is that we can still merge that feature into master and get all of
    >     > our testing (both internally and externally) running against it.   If you
    >     > stick with just branching off master and this situation arises then we get
    >     > in the spot where we just don't merge the PR at all until after the release
    >     > branch is cut (we have two examples of this right now).  I personally think
    >     > it's better to get this into master and let our automation run against it
    >     > than it is to leave it out in a PR even though it is ready.
    >     >
    >     > As for the process, I was thinking the same thing except that you don't
    >     > even need to cherry-pick.  In your scenario above you can just rebase
    >     > master into 4.x once E was merged.  This avoids the headaches of having to
    >     > cherry pick each commit while still giving us the benefit of having all of
    >     > the changes on the release branch.  We can even decide to setup automation
    >     > to rebase master into the release branch on an interval or on demand.  If
    >     > we rebase in something that turns out to be broken we just roll back to the
    >     > previous rebase (or a certain commit) until that is fixed and then rebase
    >     > again.  If a feature gets merged to master that we aren't ready to put into
    >     > the release, then we stop the automated rebase until we are ready to get it
    >     > into the release -- at that point we should consider ONLY cherry-picking
    >     > bug fixes for the remainder of that release cycle to make things easier to
    >     > rebase again.
    >     >
    >     > I am sure there will be unintended consequences, and I am sure we will have
    >     > challenges to work through, but I still think that it's worth trying.  We
    >     > should fail fast and be willing to change course if necessary.
    >     > I think if we can be consistent enough with our releases then maybe we can
    >     > discuss branching from master again.
    >     >
    >     > Thanks,
    >     > Dave
    >     >
    >     >
    >     >
    >     > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
    >     >
    >     > > I guess my concern is really about what possible reasons we might have
    >     > > to not cherry-pick a specific PR that had been merged into master
    >     > > already. The only reasons I can think of to not cherry-pick something
    >     > > would be:
    >     > > 1. the PR was completely broken (e.g. breaks the build, tests don't
    >     > > pass, doesn't compile, etc).
    >     > > 2. the PR one one of a larger multi-PR "feature", and *we know*
    >     > > releasing some parts early could cause problems (that knowing part is
    >     > > crucial)
    >     > >
    >     > > We should always strive to catch issue #1 in our CI before it merges
    >     > > into master. When things like that merge it can block people from
    >     > > doing development starting at that point until it's fixed.
    >     > > I can see why issue #2 might arise due to multiple different people
    >     > > developing one larger feature, but we should still strive to break
    >     > > larger features up into individual pieces that can reasonably stand on
    >     > > their own if possible. Due to time constraints or other reasons I can
    >     > > see that might be a little difficult.
    >     > >
    >     > > That said, I would still strive to make the 4.x release branch a
    >     > > snapshot of master at some "known good point" in time. Let's say we
    >     > > cut 4.x at commit A, then a broken commit B gets merged into master.
    >     > > If it's completely obvious that it's a broken commit, I'd wait until
    >     > > at least a commit C gets merged to fix it. Then I would cherry-pick B
    >     > > and C into 4.x at the same time. If the fix is really like a commit E
    >     > > with unrelated commits C and D in between, ideally I would still wait
    >     > > to cherry-pick C and D until a "fix" commit E was merged (to fix
    >     > > commit B), at which point I'd cherrypick commits B through E at the
    >     > > same time, in order, to avoid any possible merge conflicts.
    >     > >
    >     > > I think to make that process easier for myself I would just
    >     > > cherry-pick everything into the 4.x branch as it gets merged into
    >     > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
    >     > > commits were not fixed/completed in time, I would reset the 4.x branch
    >     > > to the last stable commit. This would mean that unstable
    >     > > commits/incomplete features could potentially block things from going
    >     > > into the release. I might count that as a positive because it would
    >     > > incentivize us to improve our CI and do our best to also keep the
    >     > > master branch as stable as possible.
    >     > >
    >     > > The end result would be the same as just continually jumping up the
    >     > > master branch from known good, stable points in time to the next,
    >     > > creating a minor release branch at each of those points.
    >     > >
    >     > > - Rawlin
    >     > >
    >     > >
    >     > >
    >     > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
    >     > > >
    >     > > > Sorry, I have taken too long to respond.
    >     > > >
    >     > > > +1000000 on better CI and Testing, I think we all agree that needs to get
    >     > > > better and is a different beast than this topic.
    >     > > >
    >     > > > As for the release branches, I am trying to follow a process similar to
    >     > > > ATS, which is usually the model we try to follow with our project.
    >     > > > While I agree that branching from master is the ideal solution, we tried
    >     > > > that and it didn't work.  I feel like we are trying to agree to do the
    >     > > same
    >     > > > thing that we already tried and didn't work.  I think we should have a
    >     > > > little freedom to commit things into master and, if they break things,
    >     > > work
    >     > > > on getting them fixed.  We also might commit PRs to master that are
    >     > > partial
    >     > > > implementations of a bigger feature.   If we are using a time based
    >     > > > strategy to cut our releases from master then we are going to be
    >     > > > continually worrying about making sure Master stays not broken.  I think
    >     > > we
    >     > > > should try to keep master deployable, but it's fine if it's not for short
    >     > > > periods of time.
    >     > > >
    >     > > > I am not trying to suggest that we are "picky" about what goes into our
    >     > > > release branch.  I think we should try to cherry-pick as much as we can
    >     > > > into our release branch once it is ready to be cherry-picked.  I think we
    >     > > > should be able to take the tip of our release branch and deploy it
    >     > > whenever
    >     > > > we want with some confidence since we know that features/bug-fixes that
    >     > > > have been cherry-picked have already been reviewed and tested against
    >     > > > master.  Yes, its more work, but it also give us the ability to have more
    >     > > > confidence in what is in our release at any given time.  Again, we should
    >     > > > be trying to cherry-pick in everything that makes sense to cherry-pick.
    >     > > >
    >     > > > I think the process that I would like to see looks like this:
    >     > > > - Cut a 4.x branch from Master
    >     > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
    >     > > branch
    >     > > > - Tag the 4.0.0 release from the 4.0.x branch
    >     > > > - New non-breaking changes and bug fixes that go into master are also
    >     > > > Cherry Picked to 4.x
    >     > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
    >     > > > are ready for a RC.
    >     > > > - Rinse and repeat for all 4.x releases
    >     > > > - Once there's a need for a major release we make a 5.x branch from
    >     > > master
    >     > > >
    >     > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
    >     > > then
    >     > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
    >     > > have
    >     > > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
    >     > > > really that bad and B) it gives us more control.
    >     > > >
    >     > > > This should allow us to deploy much quicker and with more confidence than
    >     > > > is possible today.   We can also choose to deploy from any point in time
    >     > > > from the 4.x branch with relatively high confidence.
    >     > > >
    >     > > > I'll wrap this up by saying that I don't think there's a perfect
    >     > > solution,
    >     > > > but I think we can agree on something that is worth trying at least with
    >     > > > the 4.x release life cycle.  Once we have some experience, we can figure
    >     > > > out where it is not meeting our needs and figure out how to adjust as
    >     > > > necessary.  I think we can all agree our current process is not working.
    >     > > >
    >     > > > Thanks,
    >     > > > Dave
    >     > > >
    >     > > >
    >     > > >
    >     > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
    >     > > wrote:
    >     > > >
    >     > > > > +1 on more frequent releases
    >     > > > > +1 on improving our CI. We need to get our CI to the point where we're
    >     > > > > confident deploying changes to production if the CI passes.
    >     > > > > +1 on "trunk-based" development as opposed to GitFlow
    >     > > > > -1 on cherry-picking new features. This is risky, tedious, and
    >     > > > > represents unnecessary effort that would be better spent elsewhere,
    >     > > > > like improving our CI or adding more automated tests.
    >     > > > > +1 on cutting minor release branches directly off master
    >     > > > >
    >     > > > > I think cutting minor release branches directly off master can put us
    >     > > > > in a win-win scenario. Our releases become smaller and more frequent,
    >     > > > > and we avoid the unnecessary risk and effort of cherry-picking new
    >     > > > > features onto stale release branches.
    >     > > > >
    >     > > > > This is the process that I would propose:
    >     > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
    >     > > > > 2. Develop master for 4-6 weeks.
    >     > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
    >     > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
    >     > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
    >     > > > > cherry-pick the fix from master into the latest release branch (e.g.
    >     > > > > 4.1.x), and create a 4.1.1 RC.
    >     > > > >
    >     > > > > As the 4.x release manager, I really don't want to get into the
    >     > > > > business of selecting what should and should not go into a particular
    >     > > > > release. Every 4-6 weeks, we should just cut a new minor release
    >     > > > > branch off master and take everything that was contributed during that
    >     > > > > time, as long as everything builds fine and all the tests pass. The
    >     > > > > master branch, while it doesn't always have to be perfect, should at
    >     > > > > least always be deployable. If a new feature is merged with bugs and
    >     > > > > becomes a show-stopper, it should be either immediately remedied or
    >     > > > > reverted on master while we work on a fix.
    >     > > > >
    >     > > > > - Rawlin
    >     > > > >
    >     > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
    >     > > > > <Mi...@comcast.com> wrote:
    >     > > > > >
    >     > > > > > In my opinion, in order to get to a cadence we are discussing we
    >     > > need to
    >     > > > > put a lot more work into the CI system. It has been failing
    >     > > consistently,
    >     > > > > doesn’t block PR approvals/merges and when it actually runs it does not
    >     > > > > test anything besides build and license headers.
    >     > > > > >
    >     > > > > > In the past couple of weeks we have had (off the top of my head):
    >     > > > > >
    >     > > > > > * Unit tests broken
    >     > > > > > * TO API tests broken
    >     > > > > > * Golang vendor issues
    >     > > > > > * TO Go build issues
    >     > > > > >
    >     > > > > > All were merged and broke master.
    >     > > > > >
    >     > > > > > Yes, having release branches sounds great and more aggressive cadence
    >     > > > > will minimize the amount of risk but I think that comes with the need
    >     > > to
    >     > > > > improve on our automated validation and testing.
    >     > > > > >
    >     > > > > > We have proven that we have not kept master in a good state and
    >     > > adding
    >     > > > > more releases (more branches) will make it even harder to keep that
    >     > > > > stability we are already fighting.
    >     > > > > >
    >     > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
    >     > > wrote:
    >     > > > > >
    >     > > > > >     Yeah, I get it. No one company should be driving release
    >     > > > > schedules/scope.
    >     > > > > >
    >     > > > > >     What I was really getting at is if ANY company has a recent
    >     > > version
    >     > > > > in a
    >     > > > > >     test or prod-like enviro (meaning the version is being exercised
    >     > > and
    >     > > > > >     thoroughly tested), then maybe we consider simply cutting a
    >     > > release
    >     > > > > from
    >     > > > > >     that version. For example, imagine company X had this version in
    >     > > a
    >     > > > > >     test/prod enviro:
    >     > > > > >
    >     > > > > >     Master-10287.7e62d07 (
    >     > > > > >
    >     > > > >
    >     > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
    >     > > > > )
    >     > > > > >
    >     > > > > >     Then I would be in favor of cutting the 4.0 release from
    >     > > > > >     Master-10287.7e62d07 as we know it is proven to work based on
    >     > > > > company X's
    >     > > > > >     actual use of it.
    >     > > > > >
    >     > > > > >     The truth is (imo), we don't have the bandwidth to manually
    >     > > verify
    >     > > > > releases
    >     > > > > >     that we are not using/have not used nor do we have the necessary
    >     > > > > automated
    >     > > > > >     test coverage to verify these releases. So most of our releases
    >     > > are
    >     > > > > >     significantly untested.
    >     > > > > >
    >     > > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
    >     > > > > wrote:
    >     > > > > >
    >     > > > > >     > I'm really not a fan of allowing Comcast to dictate the release
    >     > > > > scope and
    >     > > > > >     > schedule. If cherry-picking is too messy, then we can just cut
    >     > > new
    >     > > > > minor
    >     > > > > >     > releases directly from master.
    >     > > > > >     >
    >     > > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
    >     > > mitchell852@gmail.com>
    >     > > > > wrote:
    >     > > > > >     >
    >     > > > > >     > > I don't think it's as easy as cherry picking (backporting)
    >     > > > > certain
    >     > > > > >     > features
    >     > > > > >     > > into a release branch. I could be wrong but I really don't
    >     > > think
    >     > > > > it is.
    >     > > > > >     > >
    >     > > > > >     > > So what I'm hearing is that 4.0 gets cut from master and we
    >     > > go
    >     > > > > through
    >     > > > > >     > with
    >     > > > > >     > > our normal process of testing, validating, etc. At the same
    >     > > > > time, 4.1
    >     > > > > >     > > branch is created from 4.0. Master moves on as normal. Let's
    >     > > say
    >     > > > > 25
    >     > > > > >     > commits
    >     > > > > >     > > come in to master during the next 6 weeks. During that 6
    >     > > weeks,
    >     > > > > we cherry
    >     > > > > >     > > pick only the features that look interesting for 4.1? (the
    >     > > rest
    >     > > > > just stay
    >     > > > > >     > > in master and will get picked up in 5.0?)
    >     > > > > >     > >
    >     > > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
    >     > > > > weeks apart
    >     > > > > >     > and
    >     > > > > >     > > have only "interesting" features so the releases are
    >     > > > > incrementally very
    >     > > > > >     > > small and much easier to test/validate, however, those
    >     > > features
    >     > > > > might
    >     > > > > >     > > depend on other commits. At some point, cherry picking
    >     > > becomes
    >     > > > > impossible
    >     > > > > >     > > as the foundation of the code base has shifted so
    >     > > drastically.
    >     > > > > Cherry
    >     > > > > >     > > picking will become very messy and introduce a high level of
    >     > > > > risk imo.
    >     > > > > >     > >
    >     > > > > >     > > Also, you'd have to manage the scope of these minor releases
    >     > > > > which we are
    >     > > > > >     > > not very good at because the designated release manager has a
    >     > > > > full-time
    >     > > > > >     > job
    >     > > > > >     > > to attend to and now we have even more releases to
    >     > > > > >     > > scope/test/validate/approve. I like the idea of faster
    >     > > releases
    >     > > > > but this
    >     > > > > >     > > sounds like more work that nobody has the bandwidth for.
    >     > > > > >     > >
    >     > > > > >     > > Ok, so rather than shooting down an idea with no alternative
    >     > > > > proposal, I
    >     > > > > >     > > propose this. At Comcast, we are trying to get to the point
    >     > > > > where we
    >     > > > > >     > > release TC to prod from the head of master every few weeks.
    >     > > What
    >     > > > > if every
    >     > > > > >     > > time Comcast does an internal upgrade, they note the commit
    >     > > hash
    >     > > > > and once
    >     > > > > >     > > comcast is happy with the upgrade in their prod enviro, they
    >     > > > > propose a
    >     > > > > >     > > release pinned to that commit hash? With this approach you'd
    >     > > end
    >     > > > > up with:
    >     > > > > >     > >
    >     > > > > >     > > 1. fast releases (hopefully every few weeks)
    >     > > > > >     > > 2. well tested releases as it's in a real prod enviro
    >     > > > > >     > > 3. little/no validation work for anyone
    >     > > > > >     > >
    >     > > > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
    >     > > > > front at the
    >     > > > > >     > > edge of master and running it in a trusted enviro, they could
    >     > > > > propose a
    >     > > > > >     > > release as well.
    >     > > > > >     > >
    >     > > > > >     > > Jeremy
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > >
    >     > > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
    >     > > > > david.neuman64@gmail.com>
    >     > > > > >     > > wrote:
    >     > > > > >     > >
    >     > > > > >     > > > I have been thinking about how we can get better with our
    >     > > > > release
    >     > > > > >     > > cadence.
    >     > > > > >     > > > I feel like we have slowed to a crawl and not been as good
    >     > > as
    >     > > > > we should
    >     > > > > >     > > > about how we release.  Our last Major release was in March
    >     > > and
    >     > > > > we
    >     > > > > >     > haven't
    >     > > > > >     > > > had a real release since.   Moving forward I would like to
    >     > > see
    >     > > > > us get
    >     > > > > >     > on
    >     > > > > >     > > a
    >     > > > > >     > > > more consistent cadence and provide smaller releases that
    >     > > can
    >     > > > > be more
    >     > > > > >     > > > easily digested by the community.  It's my hope that we can
    >     > > > > get to a
    >     > > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
    >     > > we are
    >     > > > > >     > releasing
    >     > > > > >     > > > in such a way that not all releases all required (e.g. if
    >     > > you
    >     > > > > are on
    >     > > > > >     > 4.0
    >     > > > > >     > > > you don't need 4.1, you can just install 4.2), and we are
    >     > > more
    >     > > > > >     > deliberate
    >     > > > > >     > > > about what we put into a release.
    >     > > > > >     > > >
    >     > > > > >     > > > I think we can accomplish this by using our release
    >     > > branches
    >     > > > > better.
    >     > > > > >     > For
    >     > > > > >     > > > each major release we will cut a release branch from master
    >     > > > > (e.g. 4.x)
    >     > > > > >     > > and
    >     > > > > >     > > > then we will use cherry-picking to add new features and bug
    >     > > > > fixes to
    >     > > > > >     > that
    >     > > > > >     > > > release.  This means that if 4.0 has been released and you
    >     > > > > want to get
    >     > > > > >     > > your
    >     > > > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
    >     > > > > master and
    >     > > > > >     > then
    >     > > > > >     > > > cheery pick your squash merged commit to the 4.x branch
    >     > > which
    >     > > > > we will
    >     > > > > >     > use
    >     > > > > >     > > > to create the 4.1 release.  I think we should allow either
    >     > > the
    >     > > > > >     > > contributor
    >     > > > > >     > > > or the committer (who is merging the PR) to suggest if a
    >     > > > > feature goes
    >     > > > > >     > > into
    >     > > > > >     > > > the release branch, and if we disagree we can take it to
    >     > > the
    >     > > > > list.  If
    >     > > > > >     > we
    >     > > > > >     > > > decide that every PR to master also goes into 4.1 then so
    >     > > be
    >     > > > > it, but at
    >     > > > > >     > > > least we are making a conscious decision of what goes into
    >     > > the
    >     > > > > next
    >     > > > > >     > > > release.   This will allow us to not be so worried about
    >     > > what
    >     > > > > we are
    >     > > > > >     > > > merging into master and how that will affect our next
    >     > > release.
    >     > > > > >     > According
    >     > > > > >     > > > to our 6 week cadence we will cut a new release from the
    >     > > > > current
    >     > > > > >     > feature
    >     > > > > >     > > > branch and put that up for a vote.  These releases will be
    >     > > > > small and
    >     > > > > >     > > > testable enough that we can get feedback quickly and move
    >     > > from
    >     > > > > RC to
    >     > > > > >     > > > release in a short time.  If a release has too many issues
    >     > > > > then we may
    >     > > > > >     > > end
    >     > > > > >     > > > up deciding to skip that minor release in favor of the next
    >     > > > > one,
    >     > > > > >     > > hopefully
    >     > > > > >     > > > this does not happen very often if at all.
    >     > > > > >     > > >
    >     > > > > >     > > > Once a breaking change is introduced to master which will
    >     > > > > require a new
    >     > > > > >     > > > major release, we will create a new major release branch
    >     > > from
    >     > > > > master
    >     > > > > >     > > (which
    >     > > > > >     > > > will mean a new release manager).  We will then repeat the
    >     > > > > same process
    >     > > > > >     > > > with the new release branch, etc, etc.
    >     > > > > >     > > >
    >     > > > > >     > > > As for LTS we will provide support for the latest major
    >     > > > > release plus
    >     > > > > >     > the
    >     > > > > >     > > > one previous.  So, once we release 4.0 we will support 4.0
    >     > > and
    >     > > > > 3.1.
    >     > > > > >     > Any
    >     > > > > >     > > > security issues that arise will -- if present -- be
    >     > > applied to
    >     > > > > each of
    >     > > > > >     > > > these versions.  Once 4.1 is released we will support 4.1
    >     > > and
    >     > > > > 3.1, etc,
    >     > > > > >     > > > etc.
    >     > > > > >     > > >
    >     > > > > >     > > > Please let me know if you have any thoughts on this plan.
    >     > > If
    >     > > > > there are
    >     > > > > >     > > no
    >     > > > > >     > > > major objections I would like to try this with 4.x with the
    >     > > > > idea that
    >     > > > > >     > we
    >     > > > > >     > > > will adjust how we do things as necessary.
    >     > > > > >     > > >
    >     > > > > >     > > > We need to get better at releasing and something needs to
    >     > > > > change,
    >     > > > > >     > > hopefully
    >     > > > > >     > > > this can get us going in the right direction.
    >     > > > > >     > > >
    >     > > > > >     > > > Thanks,
    >     > > > > >     > > > Dave
    >     > > > > >     > > >
    >     > > > > >     > > >
    >     > > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
    >     > > > > cherry-picks to
    >     > > > > >     > our
    >     > > > > >     > > > release branch to control what goes into a minor release.
    >     > > > > Major
    >     > > > > >     > releases
    >     > > > > >     > > > will be cut from master as necessary - such as if there is
    >     > > a
    >     > > > > breaking
    >     > > > > >     > > > change that is introduced.
    >     > > > > >     > > >
    >     > > > > >     > >
    >     > > > > >     >
    >     > > > > >
    >     > > > > >
    >     > > > >
    >     > >
    >
    >

Re: [EXTERNAL] Re: Release Process

[Rawlin Peters <ra...@apache.org>]

I agree, but this release process wouldn't lead to long-lived feature
branches since all development would still be done on master and
merged into master. Unstable or incomplete features could potentially
delay releases from going out, but I agree that feature flags could
mitigate those problems. Anything we can do to keep the master branch
deployable is a +1 in my book.

- Rawlin

On Thu, Oct 31, 2019 at 2:46 PM Genz, Geoffrey
<Ge...@comcast.com> wrote:
>
> Just throwing out that the judicious use of feature flags could reduce the pain from unstable or incomplete new features.  Long lived feature branches are something I believe should be avoided.
>
> Geoff
>
> On 10/31/19, 2:32 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
>     Alright, I think that sounds fair. In general what I'm hearing is,
>     every 4-6 weeks:
>     1. fast-forward 4.x to the most recent "stable" commit of master
>     (maybe we automate this and fast-forward every day if some set of
>     tests pass, unless we explicitly know master is unstable for other
>     reasons)
>     2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
>     4.*.x that are necessary for the release (e.g. bugs that break the
>     data plane or prevent operators from being able to do their job or run
>     the control plane)
>
>     Which means the release manager needs to know/track:
>     - unstable commits along with their corresponding fixes
>     - incomplete features
>     - features we'd like to hold until the next release (due to current
>     release having enough already)
>     - any cherry-picks that were deemed necessary for a certain release
>     (so that they can either be carried forward into the next minor
>     release or, ideally, included as part of fast-forwarding 4.x to the
>     most recent stable commit on master)
>
>     I think this will work. We gain the ability to:
>     1. let master be "unstable" periodically
>     2. control the size of releases
>     3. make releases mirror master as closely as possible (because they
>     will just be a snapshot of the most recent version of master that was
>     considered stable -- unless absolutely necessary bug/security fixes
>     were cherry-picked in)
>
>     Does that all sound right, Dave?
>
>     - Rawlin
>
>
>
>
>
>
>
>     On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com> wrote:
>     >
>     > I think a third reason would be that we are getting very close to our
>     > release branch date, it's a large/complex feature, and we feel like the
>     > release already has enough in it.  In that case we could hold off on adding
>     > that change to the 4.x branch until after we cut the 4.x.x branch.  The
>     > benefit is that we can still merge that feature into master and get all of
>     > our testing (both internally and externally) running against it.   If you
>     > stick with just branching off master and this situation arises then we get
>     > in the spot where we just don't merge the PR at all until after the release
>     > branch is cut (we have two examples of this right now).  I personally think
>     > it's better to get this into master and let our automation run against it
>     > than it is to leave it out in a PR even though it is ready.
>     >
>     > As for the process, I was thinking the same thing except that you don't
>     > even need to cherry-pick.  In your scenario above you can just rebase
>     > master into 4.x once E was merged.  This avoids the headaches of having to
>     > cherry pick each commit while still giving us the benefit of having all of
>     > the changes on the release branch.  We can even decide to setup automation
>     > to rebase master into the release branch on an interval or on demand.  If
>     > we rebase in something that turns out to be broken we just roll back to the
>     > previous rebase (or a certain commit) until that is fixed and then rebase
>     > again.  If a feature gets merged to master that we aren't ready to put into
>     > the release, then we stop the automated rebase until we are ready to get it
>     > into the release -- at that point we should consider ONLY cherry-picking
>     > bug fixes for the remainder of that release cycle to make things easier to
>     > rebase again.
>     >
>     > I am sure there will be unintended consequences, and I am sure we will have
>     > challenges to work through, but I still think that it's worth trying.  We
>     > should fail fast and be willing to change course if necessary.
>     > I think if we can be consistent enough with our releases then maybe we can
>     > discuss branching from master again.
>     >
>     > Thanks,
>     > Dave
>     >
>     >
>     >
>     > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
>     >
>     > > I guess my concern is really about what possible reasons we might have
>     > > to not cherry-pick a specific PR that had been merged into master
>     > > already. The only reasons I can think of to not cherry-pick something
>     > > would be:
>     > > 1. the PR was completely broken (e.g. breaks the build, tests don't
>     > > pass, doesn't compile, etc).
>     > > 2. the PR one one of a larger multi-PR "feature", and *we know*
>     > > releasing some parts early could cause problems (that knowing part is
>     > > crucial)
>     > >
>     > > We should always strive to catch issue #1 in our CI before it merges
>     > > into master. When things like that merge it can block people from
>     > > doing development starting at that point until it's fixed.
>     > > I can see why issue #2 might arise due to multiple different people
>     > > developing one larger feature, but we should still strive to break
>     > > larger features up into individual pieces that can reasonably stand on
>     > > their own if possible. Due to time constraints or other reasons I can
>     > > see that might be a little difficult.
>     > >
>     > > That said, I would still strive to make the 4.x release branch a
>     > > snapshot of master at some "known good point" in time. Let's say we
>     > > cut 4.x at commit A, then a broken commit B gets merged into master.
>     > > If it's completely obvious that it's a broken commit, I'd wait until
>     > > at least a commit C gets merged to fix it. Then I would cherry-pick B
>     > > and C into 4.x at the same time. If the fix is really like a commit E
>     > > with unrelated commits C and D in between, ideally I would still wait
>     > > to cherry-pick C and D until a "fix" commit E was merged (to fix
>     > > commit B), at which point I'd cherrypick commits B through E at the
>     > > same time, in order, to avoid any possible merge conflicts.
>     > >
>     > > I think to make that process easier for myself I would just
>     > > cherry-pick everything into the 4.x branch as it gets merged into
>     > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
>     > > commits were not fixed/completed in time, I would reset the 4.x branch
>     > > to the last stable commit. This would mean that unstable
>     > > commits/incomplete features could potentially block things from going
>     > > into the release. I might count that as a positive because it would
>     > > incentivize us to improve our CI and do our best to also keep the
>     > > master branch as stable as possible.
>     > >
>     > > The end result would be the same as just continually jumping up the
>     > > master branch from known good, stable points in time to the next,
>     > > creating a minor release branch at each of those points.
>     > >
>     > > - Rawlin
>     > >
>     > >
>     > >
>     > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
>     > > >
>     > > > Sorry, I have taken too long to respond.
>     > > >
>     > > > +1000000 on better CI and Testing, I think we all agree that needs to get
>     > > > better and is a different beast than this topic.
>     > > >
>     > > > As for the release branches, I am trying to follow a process similar to
>     > > > ATS, which is usually the model we try to follow with our project.
>     > > > While I agree that branching from master is the ideal solution, we tried
>     > > > that and it didn't work.  I feel like we are trying to agree to do the
>     > > same
>     > > > thing that we already tried and didn't work.  I think we should have a
>     > > > little freedom to commit things into master and, if they break things,
>     > > work
>     > > > on getting them fixed.  We also might commit PRs to master that are
>     > > partial
>     > > > implementations of a bigger feature.   If we are using a time based
>     > > > strategy to cut our releases from master then we are going to be
>     > > > continually worrying about making sure Master stays not broken.  I think
>     > > we
>     > > > should try to keep master deployable, but it's fine if it's not for short
>     > > > periods of time.
>     > > >
>     > > > I am not trying to suggest that we are "picky" about what goes into our
>     > > > release branch.  I think we should try to cherry-pick as much as we can
>     > > > into our release branch once it is ready to be cherry-picked.  I think we
>     > > > should be able to take the tip of our release branch and deploy it
>     > > whenever
>     > > > we want with some confidence since we know that features/bug-fixes that
>     > > > have been cherry-picked have already been reviewed and tested against
>     > > > master.  Yes, its more work, but it also give us the ability to have more
>     > > > confidence in what is in our release at any given time.  Again, we should
>     > > > be trying to cherry-pick in everything that makes sense to cherry-pick.
>     > > >
>     > > > I think the process that I would like to see looks like this:
>     > > > - Cut a 4.x branch from Master
>     > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
>     > > branch
>     > > > - Tag the 4.0.0 release from the 4.0.x branch
>     > > > - New non-breaking changes and bug fixes that go into master are also
>     > > > Cherry Picked to 4.x
>     > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
>     > > > are ready for a RC.
>     > > > - Rinse and repeat for all 4.x releases
>     > > > - Once there's a need for a major release we make a 5.x branch from
>     > > master
>     > > >
>     > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
>     > > then
>     > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
>     > > have
>     > > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
>     > > > really that bad and B) it gives us more control.
>     > > >
>     > > > This should allow us to deploy much quicker and with more confidence than
>     > > > is possible today.   We can also choose to deploy from any point in time
>     > > > from the 4.x branch with relatively high confidence.
>     > > >
>     > > > I'll wrap this up by saying that I don't think there's a perfect
>     > > solution,
>     > > > but I think we can agree on something that is worth trying at least with
>     > > > the 4.x release life cycle.  Once we have some experience, we can figure
>     > > > out where it is not meeting our needs and figure out how to adjust as
>     > > > necessary.  I think we can all agree our current process is not working.
>     > > >
>     > > > Thanks,
>     > > > Dave
>     > > >
>     > > >
>     > > >
>     > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
>     > > wrote:
>     > > >
>     > > > > +1 on more frequent releases
>     > > > > +1 on improving our CI. We need to get our CI to the point where we're
>     > > > > confident deploying changes to production if the CI passes.
>     > > > > +1 on "trunk-based" development as opposed to GitFlow
>     > > > > -1 on cherry-picking new features. This is risky, tedious, and
>     > > > > represents unnecessary effort that would be better spent elsewhere,
>     > > > > like improving our CI or adding more automated tests.
>     > > > > +1 on cutting minor release branches directly off master
>     > > > >
>     > > > > I think cutting minor release branches directly off master can put us
>     > > > > in a win-win scenario. Our releases become smaller and more frequent,
>     > > > > and we avoid the unnecessary risk and effort of cherry-picking new
>     > > > > features onto stale release branches.
>     > > > >
>     > > > > This is the process that I would propose:
>     > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
>     > > > > 2. Develop master for 4-6 weeks.
>     > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
>     > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
>     > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
>     > > > > cherry-pick the fix from master into the latest release branch (e.g.
>     > > > > 4.1.x), and create a 4.1.1 RC.
>     > > > >
>     > > > > As the 4.x release manager, I really don't want to get into the
>     > > > > business of selecting what should and should not go into a particular
>     > > > > release. Every 4-6 weeks, we should just cut a new minor release
>     > > > > branch off master and take everything that was contributed during that
>     > > > > time, as long as everything builds fine and all the tests pass. The
>     > > > > master branch, while it doesn't always have to be perfect, should at
>     > > > > least always be deployable. If a new feature is merged with bugs and
>     > > > > becomes a show-stopper, it should be either immediately remedied or
>     > > > > reverted on master while we work on a fix.
>     > > > >
>     > > > > - Rawlin
>     > > > >
>     > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
>     > > > > <Mi...@comcast.com> wrote:
>     > > > > >
>     > > > > > In my opinion, in order to get to a cadence we are discussing we
>     > > need to
>     > > > > put a lot more work into the CI system. It has been failing
>     > > consistently,
>     > > > > doesn’t block PR approvals/merges and when it actually runs it does not
>     > > > > test anything besides build and license headers.
>     > > > > >
>     > > > > > In the past couple of weeks we have had (off the top of my head):
>     > > > > >
>     > > > > > * Unit tests broken
>     > > > > > * TO API tests broken
>     > > > > > * Golang vendor issues
>     > > > > > * TO Go build issues
>     > > > > >
>     > > > > > All were merged and broke master.
>     > > > > >
>     > > > > > Yes, having release branches sounds great and more aggressive cadence
>     > > > > will minimize the amount of risk but I think that comes with the need
>     > > to
>     > > > > improve on our automated validation and testing.
>     > > > > >
>     > > > > > We have proven that we have not kept master in a good state and
>     > > adding
>     > > > > more releases (more branches) will make it even harder to keep that
>     > > > > stability we are already fighting.
>     > > > > >
>     > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
>     > > wrote:
>     > > > > >
>     > > > > >     Yeah, I get it. No one company should be driving release
>     > > > > schedules/scope.
>     > > > > >
>     > > > > >     What I was really getting at is if ANY company has a recent
>     > > version
>     > > > > in a
>     > > > > >     test or prod-like enviro (meaning the version is being exercised
>     > > and
>     > > > > >     thoroughly tested), then maybe we consider simply cutting a
>     > > release
>     > > > > from
>     > > > > >     that version. For example, imagine company X had this version in
>     > > a
>     > > > > >     test/prod enviro:
>     > > > > >
>     > > > > >     Master-10287.7e62d07 (
>     > > > > >
>     > > > >
>     > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
>     > > > > )
>     > > > > >
>     > > > > >     Then I would be in favor of cutting the 4.0 release from
>     > > > > >     Master-10287.7e62d07 as we know it is proven to work based on
>     > > > > company X's
>     > > > > >     actual use of it.
>     > > > > >
>     > > > > >     The truth is (imo), we don't have the bandwidth to manually
>     > > verify
>     > > > > releases
>     > > > > >     that we are not using/have not used nor do we have the necessary
>     > > > > automated
>     > > > > >     test coverage to verify these releases. So most of our releases
>     > > are
>     > > > > >     significantly untested.
>     > > > > >
>     > > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
>     > > > > wrote:
>     > > > > >
>     > > > > >     > I'm really not a fan of allowing Comcast to dictate the release
>     > > > > scope and
>     > > > > >     > schedule. If cherry-picking is too messy, then we can just cut
>     > > new
>     > > > > minor
>     > > > > >     > releases directly from master.
>     > > > > >     >
>     > > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
>     > > mitchell852@gmail.com>
>     > > > > wrote:
>     > > > > >     >
>     > > > > >     > > I don't think it's as easy as cherry picking (backporting)
>     > > > > certain
>     > > > > >     > features
>     > > > > >     > > into a release branch. I could be wrong but I really don't
>     > > think
>     > > > > it is.
>     > > > > >     > >
>     > > > > >     > > So what I'm hearing is that 4.0 gets cut from master and we
>     > > go
>     > > > > through
>     > > > > >     > with
>     > > > > >     > > our normal process of testing, validating, etc. At the same
>     > > > > time, 4.1
>     > > > > >     > > branch is created from 4.0. Master moves on as normal. Let's
>     > > say
>     > > > > 25
>     > > > > >     > commits
>     > > > > >     > > come in to master during the next 6 weeks. During that 6
>     > > weeks,
>     > > > > we cherry
>     > > > > >     > > pick only the features that look interesting for 4.1? (the
>     > > rest
>     > > > > just stay
>     > > > > >     > > in master and will get picked up in 5.0?)
>     > > > > >     > >
>     > > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
>     > > > > weeks apart
>     > > > > >     > and
>     > > > > >     > > have only "interesting" features so the releases are
>     > > > > incrementally very
>     > > > > >     > > small and much easier to test/validate, however, those
>     > > features
>     > > > > might
>     > > > > >     > > depend on other commits. At some point, cherry picking
>     > > becomes
>     > > > > impossible
>     > > > > >     > > as the foundation of the code base has shifted so
>     > > drastically.
>     > > > > Cherry
>     > > > > >     > > picking will become very messy and introduce a high level of
>     > > > > risk imo.
>     > > > > >     > >
>     > > > > >     > > Also, you'd have to manage the scope of these minor releases
>     > > > > which we are
>     > > > > >     > > not very good at because the designated release manager has a
>     > > > > full-time
>     > > > > >     > job
>     > > > > >     > > to attend to and now we have even more releases to
>     > > > > >     > > scope/test/validate/approve. I like the idea of faster
>     > > releases
>     > > > > but this
>     > > > > >     > > sounds like more work that nobody has the bandwidth for.
>     > > > > >     > >
>     > > > > >     > > Ok, so rather than shooting down an idea with no alternative
>     > > > > proposal, I
>     > > > > >     > > propose this. At Comcast, we are trying to get to the point
>     > > > > where we
>     > > > > >     > > release TC to prod from the head of master every few weeks.
>     > > What
>     > > > > if every
>     > > > > >     > > time Comcast does an internal upgrade, they note the commit
>     > > hash
>     > > > > and once
>     > > > > >     > > comcast is happy with the upgrade in their prod enviro, they
>     > > > > propose a
>     > > > > >     > > release pinned to that commit hash? With this approach you'd
>     > > end
>     > > > > up with:
>     > > > > >     > >
>     > > > > >     > > 1. fast releases (hopefully every few weeks)
>     > > > > >     > > 2. well tested releases as it's in a real prod enviro
>     > > > > >     > > 3. little/no validation work for anyone
>     > > > > >     > >
>     > > > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
>     > > > > front at the
>     > > > > >     > > edge of master and running it in a trusted enviro, they could
>     > > > > propose a
>     > > > > >     > > release as well.
>     > > > > >     > >
>     > > > > >     > > Jeremy
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > >
>     > > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
>     > > > > david.neuman64@gmail.com>
>     > > > > >     > > wrote:
>     > > > > >     > >
>     > > > > >     > > > I have been thinking about how we can get better with our
>     > > > > release
>     > > > > >     > > cadence.
>     > > > > >     > > > I feel like we have slowed to a crawl and not been as good
>     > > as
>     > > > > we should
>     > > > > >     > > > about how we release.  Our last Major release was in March
>     > > and
>     > > > > we
>     > > > > >     > haven't
>     > > > > >     > > > had a real release since.   Moving forward I would like to
>     > > see
>     > > > > us get
>     > > > > >     > on
>     > > > > >     > > a
>     > > > > >     > > > more consistent cadence and provide smaller releases that
>     > > can
>     > > > > be more
>     > > > > >     > > > easily digested by the community.  It's my hope that we can
>     > > > > get to a
>     > > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
>     > > we are
>     > > > > >     > releasing
>     > > > > >     > > > in such a way that not all releases all required (e.g. if
>     > > you
>     > > > > are on
>     > > > > >     > 4.0
>     > > > > >     > > > you don't need 4.1, you can just install 4.2), and we are
>     > > more
>     > > > > >     > deliberate
>     > > > > >     > > > about what we put into a release.
>     > > > > >     > > >
>     > > > > >     > > > I think we can accomplish this by using our release
>     > > branches
>     > > > > better.
>     > > > > >     > For
>     > > > > >     > > > each major release we will cut a release branch from master
>     > > > > (e.g. 4.x)
>     > > > > >     > > and
>     > > > > >     > > > then we will use cherry-picking to add new features and bug
>     > > > > fixes to
>     > > > > >     > that
>     > > > > >     > > > release.  This means that if 4.0 has been released and you
>     > > > > want to get
>     > > > > >     > > your
>     > > > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
>     > > > > master and
>     > > > > >     > then
>     > > > > >     > > > cheery pick your squash merged commit to the 4.x branch
>     > > which
>     > > > > we will
>     > > > > >     > use
>     > > > > >     > > > to create the 4.1 release.  I think we should allow either
>     > > the
>     > > > > >     > > contributor
>     > > > > >     > > > or the committer (who is merging the PR) to suggest if a
>     > > > > feature goes
>     > > > > >     > > into
>     > > > > >     > > > the release branch, and if we disagree we can take it to
>     > > the
>     > > > > list.  If
>     > > > > >     > we
>     > > > > >     > > > decide that every PR to master also goes into 4.1 then so
>     > > be
>     > > > > it, but at
>     > > > > >     > > > least we are making a conscious decision of what goes into
>     > > the
>     > > > > next
>     > > > > >     > > > release.   This will allow us to not be so worried about
>     > > what
>     > > > > we are
>     > > > > >     > > > merging into master and how that will affect our next
>     > > release.
>     > > > > >     > According
>     > > > > >     > > > to our 6 week cadence we will cut a new release from the
>     > > > > current
>     > > > > >     > feature
>     > > > > >     > > > branch and put that up for a vote.  These releases will be
>     > > > > small and
>     > > > > >     > > > testable enough that we can get feedback quickly and move
>     > > from
>     > > > > RC to
>     > > > > >     > > > release in a short time.  If a release has too many issues
>     > > > > then we may
>     > > > > >     > > end
>     > > > > >     > > > up deciding to skip that minor release in favor of the next
>     > > > > one,
>     > > > > >     > > hopefully
>     > > > > >     > > > this does not happen very often if at all.
>     > > > > >     > > >
>     > > > > >     > > > Once a breaking change is introduced to master which will
>     > > > > require a new
>     > > > > >     > > > major release, we will create a new major release branch
>     > > from
>     > > > > master
>     > > > > >     > > (which
>     > > > > >     > > > will mean a new release manager).  We will then repeat the
>     > > > > same process
>     > > > > >     > > > with the new release branch, etc, etc.
>     > > > > >     > > >
>     > > > > >     > > > As for LTS we will provide support for the latest major
>     > > > > release plus
>     > > > > >     > the
>     > > > > >     > > > one previous.  So, once we release 4.0 we will support 4.0
>     > > and
>     > > > > 3.1.
>     > > > > >     > Any
>     > > > > >     > > > security issues that arise will -- if present -- be
>     > > applied to
>     > > > > each of
>     > > > > >     > > > these versions.  Once 4.1 is released we will support 4.1
>     > > and
>     > > > > 3.1, etc,
>     > > > > >     > > > etc.
>     > > > > >     > > >
>     > > > > >     > > > Please let me know if you have any thoughts on this plan.
>     > > If
>     > > > > there are
>     > > > > >     > > no
>     > > > > >     > > > major objections I would like to try this with 4.x with the
>     > > > > idea that
>     > > > > >     > we
>     > > > > >     > > > will adjust how we do things as necessary.
>     > > > > >     > > >
>     > > > > >     > > > We need to get better at releasing and something needs to
>     > > > > change,
>     > > > > >     > > hopefully
>     > > > > >     > > > this can get us going in the right direction.
>     > > > > >     > > >
>     > > > > >     > > > Thanks,
>     > > > > >     > > > Dave
>     > > > > >     > > >
>     > > > > >     > > >
>     > > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
>     > > > > cherry-picks to
>     > > > > >     > our
>     > > > > >     > > > release branch to control what goes into a minor release.
>     > > > > Major
>     > > > > >     > releases
>     > > > > >     > > > will be cut from master as necessary - such as if there is
>     > > a
>     > > > > breaking
>     > > > > >     > > > change that is introduced.
>     > > > > >     > > >
>     > > > > >     > >
>     > > > > >     >
>     > > > > >
>     > > > > >
>     > > > >
>     > >
>
>

Re: [EXTERNAL] Re: Release Process

["Genz, Geoffrey" <Ge...@comcast.com>]

Just throwing out that the judicious use of feature flags could reduce the pain from unstable or incomplete new features.  Long lived feature branches are something I believe should be avoided.

Geoff

On 10/31/19, 2:32 PM, "Rawlin Peters" <ra...@apache.org> wrote:

    Alright, I think that sounds fair. In general what I'm hearing is,
    every 4-6 weeks:
    1. fast-forward 4.x to the most recent "stable" commit of master
    (maybe we automate this and fast-forward every day if some set of
    tests pass, unless we explicitly know master is unstable for other
    reasons)
    2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
    4.*.x that are necessary for the release (e.g. bugs that break the
    data plane or prevent operators from being able to do their job or run
    the control plane)

    Which means the release manager needs to know/track:
    - unstable commits along with their corresponding fixes
    - incomplete features
    - features we'd like to hold until the next release (due to current
    release having enough already)
    - any cherry-picks that were deemed necessary for a certain release
    (so that they can either be carried forward into the next minor
    release or, ideally, included as part of fast-forwarding 4.x to the
    most recent stable commit on master)

    I think this will work. We gain the ability to:
    1. let master be "unstable" periodically
    2. control the size of releases
    3. make releases mirror master as closely as possible (because they
    will just be a snapshot of the most recent version of master that was
    considered stable -- unless absolutely necessary bug/security fixes
    were cherry-picked in)

    Does that all sound right, Dave?

    - Rawlin







    On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com> wrote:
    >
    > I think a third reason would be that we are getting very close to our
    > release branch date, it's a large/complex feature, and we feel like the
    > release already has enough in it.  In that case we could hold off on adding
    > that change to the 4.x branch until after we cut the 4.x.x branch.  The
    > benefit is that we can still merge that feature into master and get all of
    > our testing (both internally and externally) running against it.   If you
    > stick with just branching off master and this situation arises then we get
    > in the spot where we just don't merge the PR at all until after the release
    > branch is cut (we have two examples of this right now).  I personally think
    > it's better to get this into master and let our automation run against it
    > than it is to leave it out in a PR even though it is ready.
    >
    > As for the process, I was thinking the same thing except that you don't
    > even need to cherry-pick.  In your scenario above you can just rebase
    > master into 4.x once E was merged.  This avoids the headaches of having to
    > cherry pick each commit while still giving us the benefit of having all of
    > the changes on the release branch.  We can even decide to setup automation
    > to rebase master into the release branch on an interval or on demand.  If
    > we rebase in something that turns out to be broken we just roll back to the
    > previous rebase (or a certain commit) until that is fixed and then rebase
    > again.  If a feature gets merged to master that we aren't ready to put into
    > the release, then we stop the automated rebase until we are ready to get it
    > into the release -- at that point we should consider ONLY cherry-picking
    > bug fixes for the remainder of that release cycle to make things easier to
    > rebase again.
    >
    > I am sure there will be unintended consequences, and I am sure we will have
    > challenges to work through, but I still think that it's worth trying.  We
    > should fail fast and be willing to change course if necessary.
    > I think if we can be consistent enough with our releases then maybe we can
    > discuss branching from master again.
    >
    > Thanks,
    > Dave
    >
    >
    >
    > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
    >
    > > I guess my concern is really about what possible reasons we might have
    > > to not cherry-pick a specific PR that had been merged into master
    > > already. The only reasons I can think of to not cherry-pick something
    > > would be:
    > > 1. the PR was completely broken (e.g. breaks the build, tests don't
    > > pass, doesn't compile, etc).
    > > 2. the PR one one of a larger multi-PR "feature", and *we know*
    > > releasing some parts early could cause problems (that knowing part is
    > > crucial)
    > >
    > > We should always strive to catch issue #1 in our CI before it merges
    > > into master. When things like that merge it can block people from
    > > doing development starting at that point until it's fixed.
    > > I can see why issue #2 might arise due to multiple different people
    > > developing one larger feature, but we should still strive to break
    > > larger features up into individual pieces that can reasonably stand on
    > > their own if possible. Due to time constraints or other reasons I can
    > > see that might be a little difficult.
    > >
    > > That said, I would still strive to make the 4.x release branch a
    > > snapshot of master at some "known good point" in time. Let's say we
    > > cut 4.x at commit A, then a broken commit B gets merged into master.
    > > If it's completely obvious that it's a broken commit, I'd wait until
    > > at least a commit C gets merged to fix it. Then I would cherry-pick B
    > > and C into 4.x at the same time. If the fix is really like a commit E
    > > with unrelated commits C and D in between, ideally I would still wait
    > > to cherry-pick C and D until a "fix" commit E was merged (to fix
    > > commit B), at which point I'd cherrypick commits B through E at the
    > > same time, in order, to avoid any possible merge conflicts.
    > >
    > > I think to make that process easier for myself I would just
    > > cherry-pick everything into the 4.x branch as it gets merged into
    > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
    > > commits were not fixed/completed in time, I would reset the 4.x branch
    > > to the last stable commit. This would mean that unstable
    > > commits/incomplete features could potentially block things from going
    > > into the release. I might count that as a positive because it would
    > > incentivize us to improve our CI and do our best to also keep the
    > > master branch as stable as possible.
    > >
    > > The end result would be the same as just continually jumping up the
    > > master branch from known good, stable points in time to the next,
    > > creating a minor release branch at each of those points.
    > >
    > > - Rawlin
    > >
    > >
    > >
    > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
    > > >
    > > > Sorry, I have taken too long to respond.
    > > >
    > > > +1000000 on better CI and Testing, I think we all agree that needs to get
    > > > better and is a different beast than this topic.
    > > >
    > > > As for the release branches, I am trying to follow a process similar to
    > > > ATS, which is usually the model we try to follow with our project.
    > > > While I agree that branching from master is the ideal solution, we tried
    > > > that and it didn't work.  I feel like we are trying to agree to do the
    > > same
    > > > thing that we already tried and didn't work.  I think we should have a
    > > > little freedom to commit things into master and, if they break things,
    > > work
    > > > on getting them fixed.  We also might commit PRs to master that are
    > > partial
    > > > implementations of a bigger feature.   If we are using a time based
    > > > strategy to cut our releases from master then we are going to be
    > > > continually worrying about making sure Master stays not broken.  I think
    > > we
    > > > should try to keep master deployable, but it's fine if it's not for short
    > > > periods of time.
    > > >
    > > > I am not trying to suggest that we are "picky" about what goes into our
    > > > release branch.  I think we should try to cherry-pick as much as we can
    > > > into our release branch once it is ready to be cherry-picked.  I think we
    > > > should be able to take the tip of our release branch and deploy it
    > > whenever
    > > > we want with some confidence since we know that features/bug-fixes that
    > > > have been cherry-picked have already been reviewed and tested against
    > > > master.  Yes, its more work, but it also give us the ability to have more
    > > > confidence in what is in our release at any given time.  Again, we should
    > > > be trying to cherry-pick in everything that makes sense to cherry-pick.
    > > >
    > > > I think the process that I would like to see looks like this:
    > > > - Cut a 4.x branch from Master
    > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
    > > branch
    > > > - Tag the 4.0.0 release from the 4.0.x branch
    > > > - New non-breaking changes and bug fixes that go into master are also
    > > > Cherry Picked to 4.x
    > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
    > > > are ready for a RC.
    > > > - Rinse and repeat for all 4.x releases
    > > > - Once there's a need for a major release we make a 5.x branch from
    > > master
    > > >
    > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
    > > then
    > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
    > > have
    > > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
    > > > really that bad and B) it gives us more control.
    > > >
    > > > This should allow us to deploy much quicker and with more confidence than
    > > > is possible today.   We can also choose to deploy from any point in time
    > > > from the 4.x branch with relatively high confidence.
    > > >
    > > > I'll wrap this up by saying that I don't think there's a perfect
    > > solution,
    > > > but I think we can agree on something that is worth trying at least with
    > > > the 4.x release life cycle.  Once we have some experience, we can figure
    > > > out where it is not meeting our needs and figure out how to adjust as
    > > > necessary.  I think we can all agree our current process is not working.
    > > >
    > > > Thanks,
    > > > Dave
    > > >
    > > >
    > > >
    > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
    > > wrote:
    > > >
    > > > > +1 on more frequent releases
    > > > > +1 on improving our CI. We need to get our CI to the point where we're
    > > > > confident deploying changes to production if the CI passes.
    > > > > +1 on "trunk-based" development as opposed to GitFlow
    > > > > -1 on cherry-picking new features. This is risky, tedious, and
    > > > > represents unnecessary effort that would be better spent elsewhere,
    > > > > like improving our CI or adding more automated tests.
    > > > > +1 on cutting minor release branches directly off master
    > > > >
    > > > > I think cutting minor release branches directly off master can put us
    > > > > in a win-win scenario. Our releases become smaller and more frequent,
    > > > > and we avoid the unnecessary risk and effort of cherry-picking new
    > > > > features onto stale release branches.
    > > > >
    > > > > This is the process that I would propose:
    > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
    > > > > 2. Develop master for 4-6 weeks.
    > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
    > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
    > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
    > > > > cherry-pick the fix from master into the latest release branch (e.g.
    > > > > 4.1.x), and create a 4.1.1 RC.
    > > > >
    > > > > As the 4.x release manager, I really don't want to get into the
    > > > > business of selecting what should and should not go into a particular
    > > > > release. Every 4-6 weeks, we should just cut a new minor release
    > > > > branch off master and take everything that was contributed during that
    > > > > time, as long as everything builds fine and all the tests pass. The
    > > > > master branch, while it doesn't always have to be perfect, should at
    > > > > least always be deployable. If a new feature is merged with bugs and
    > > > > becomes a show-stopper, it should be either immediately remedied or
    > > > > reverted on master while we work on a fix.
    > > > >
    > > > > - Rawlin
    > > > >
    > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
    > > > > <Mi...@comcast.com> wrote:
    > > > > >
    > > > > > In my opinion, in order to get to a cadence we are discussing we
    > > need to
    > > > > put a lot more work into the CI system. It has been failing
    > > consistently,
    > > > > doesn’t block PR approvals/merges and when it actually runs it does not
    > > > > test anything besides build and license headers.
    > > > > >
    > > > > > In the past couple of weeks we have had (off the top of my head):
    > > > > >
    > > > > > * Unit tests broken
    > > > > > * TO API tests broken
    > > > > > * Golang vendor issues
    > > > > > * TO Go build issues
    > > > > >
    > > > > > All were merged and broke master.
    > > > > >
    > > > > > Yes, having release branches sounds great and more aggressive cadence
    > > > > will minimize the amount of risk but I think that comes with the need
    > > to
    > > > > improve on our automated validation and testing.
    > > > > >
    > > > > > We have proven that we have not kept master in a good state and
    > > adding
    > > > > more releases (more branches) will make it even harder to keep that
    > > > > stability we are already fighting.
    > > > > >
    > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
    > > wrote:
    > > > > >
    > > > > >     Yeah, I get it. No one company should be driving release
    > > > > schedules/scope.
    > > > > >
    > > > > >     What I was really getting at is if ANY company has a recent
    > > version
    > > > > in a
    > > > > >     test or prod-like enviro (meaning the version is being exercised
    > > and
    > > > > >     thoroughly tested), then maybe we consider simply cutting a
    > > release
    > > > > from
    > > > > >     that version. For example, imagine company X had this version in
    > > a
    > > > > >     test/prod enviro:
    > > > > >
    > > > > >     Master-10287.7e62d07 (
    > > > > >
    > > > >
    > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
    > > > > )
    > > > > >
    > > > > >     Then I would be in favor of cutting the 4.0 release from
    > > > > >     Master-10287.7e62d07 as we know it is proven to work based on
    > > > > company X's
    > > > > >     actual use of it.
    > > > > >
    > > > > >     The truth is (imo), we don't have the bandwidth to manually
    > > verify
    > > > > releases
    > > > > >     that we are not using/have not used nor do we have the necessary
    > > > > automated
    > > > > >     test coverage to verify these releases. So most of our releases
    > > are
    > > > > >     significantly untested.
    > > > > >
    > > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
    > > > > wrote:
    > > > > >
    > > > > >     > I'm really not a fan of allowing Comcast to dictate the release
    > > > > scope and
    > > > > >     > schedule. If cherry-picking is too messy, then we can just cut
    > > new
    > > > > minor
    > > > > >     > releases directly from master.
    > > > > >     >
    > > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
    > > mitchell852@gmail.com>
    > > > > wrote:
    > > > > >     >
    > > > > >     > > I don't think it's as easy as cherry picking (backporting)
    > > > > certain
    > > > > >     > features
    > > > > >     > > into a release branch. I could be wrong but I really don't
    > > think
    > > > > it is.
    > > > > >     > >
    > > > > >     > > So what I'm hearing is that 4.0 gets cut from master and we
    > > go
    > > > > through
    > > > > >     > with
    > > > > >     > > our normal process of testing, validating, etc. At the same
    > > > > time, 4.1
    > > > > >     > > branch is created from 4.0. Master moves on as normal. Let's
    > > say
    > > > > 25
    > > > > >     > commits
    > > > > >     > > come in to master during the next 6 weeks. During that 6
    > > weeks,
    > > > > we cherry
    > > > > >     > > pick only the features that look interesting for 4.1? (the
    > > rest
    > > > > just stay
    > > > > >     > > in master and will get picked up in 5.0?)
    > > > > >     > >
    > > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
    > > > > weeks apart
    > > > > >     > and
    > > > > >     > > have only "interesting" features so the releases are
    > > > > incrementally very
    > > > > >     > > small and much easier to test/validate, however, those
    > > features
    > > > > might
    > > > > >     > > depend on other commits. At some point, cherry picking
    > > becomes
    > > > > impossible
    > > > > >     > > as the foundation of the code base has shifted so
    > > drastically.
    > > > > Cherry
    > > > > >     > > picking will become very messy and introduce a high level of
    > > > > risk imo.
    > > > > >     > >
    > > > > >     > > Also, you'd have to manage the scope of these minor releases
    > > > > which we are
    > > > > >     > > not very good at because the designated release manager has a
    > > > > full-time
    > > > > >     > job
    > > > > >     > > to attend to and now we have even more releases to
    > > > > >     > > scope/test/validate/approve. I like the idea of faster
    > > releases
    > > > > but this
    > > > > >     > > sounds like more work that nobody has the bandwidth for.
    > > > > >     > >
    > > > > >     > > Ok, so rather than shooting down an idea with no alternative
    > > > > proposal, I
    > > > > >     > > propose this. At Comcast, we are trying to get to the point
    > > > > where we
    > > > > >     > > release TC to prod from the head of master every few weeks.
    > > What
    > > > > if every
    > > > > >     > > time Comcast does an internal upgrade, they note the commit
    > > hash
    > > > > and once
    > > > > >     > > comcast is happy with the upgrade in their prod enviro, they
    > > > > propose a
    > > > > >     > > release pinned to that commit hash? With this approach you'd
    > > end
    > > > > up with:
    > > > > >     > >
    > > > > >     > > 1. fast releases (hopefully every few weeks)
    > > > > >     > > 2. well tested releases as it's in a real prod enviro
    > > > > >     > > 3. little/no validation work for anyone
    > > > > >     > >
    > > > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
    > > > > front at the
    > > > > >     > > edge of master and running it in a trusted enviro, they could
    > > > > propose a
    > > > > >     > > release as well.
    > > > > >     > >
    > > > > >     > > Jeremy
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > >
    > > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
    > > > > david.neuman64@gmail.com>
    > > > > >     > > wrote:
    > > > > >     > >
    > > > > >     > > > I have been thinking about how we can get better with our
    > > > > release
    > > > > >     > > cadence.
    > > > > >     > > > I feel like we have slowed to a crawl and not been as good
    > > as
    > > > > we should
    > > > > >     > > > about how we release.  Our last Major release was in March
    > > and
    > > > > we
    > > > > >     > haven't
    > > > > >     > > > had a real release since.   Moving forward I would like to
    > > see
    > > > > us get
    > > > > >     > on
    > > > > >     > > a
    > > > > >     > > > more consistent cadence and provide smaller releases that
    > > can
    > > > > be more
    > > > > >     > > > easily digested by the community.  It's my hope that we can
    > > > > get to a
    > > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
    > > we are
    > > > > >     > releasing
    > > > > >     > > > in such a way that not all releases all required (e.g. if
    > > you
    > > > > are on
    > > > > >     > 4.0
    > > > > >     > > > you don't need 4.1, you can just install 4.2), and we are
    > > more
    > > > > >     > deliberate
    > > > > >     > > > about what we put into a release.
    > > > > >     > > >
    > > > > >     > > > I think we can accomplish this by using our release
    > > branches
    > > > > better.
    > > > > >     > For
    > > > > >     > > > each major release we will cut a release branch from master
    > > > > (e.g. 4.x)
    > > > > >     > > and
    > > > > >     > > > then we will use cherry-picking to add new features and bug
    > > > > fixes to
    > > > > >     > that
    > > > > >     > > > release.  This means that if 4.0 has been released and you
    > > > > want to get
    > > > > >     > > your
    > > > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
    > > > > master and
    > > > > >     > then
    > > > > >     > > > cheery pick your squash merged commit to the 4.x branch
    > > which
    > > > > we will
    > > > > >     > use
    > > > > >     > > > to create the 4.1 release.  I think we should allow either
    > > the
    > > > > >     > > contributor
    > > > > >     > > > or the committer (who is merging the PR) to suggest if a
    > > > > feature goes
    > > > > >     > > into
    > > > > >     > > > the release branch, and if we disagree we can take it to
    > > the
    > > > > list.  If
    > > > > >     > we
    > > > > >     > > > decide that every PR to master also goes into 4.1 then so
    > > be
    > > > > it, but at
    > > > > >     > > > least we are making a conscious decision of what goes into
    > > the
    > > > > next
    > > > > >     > > > release.   This will allow us to not be so worried about
    > > what
    > > > > we are
    > > > > >     > > > merging into master and how that will affect our next
    > > release.
    > > > > >     > According
    > > > > >     > > > to our 6 week cadence we will cut a new release from the
    > > > > current
    > > > > >     > feature
    > > > > >     > > > branch and put that up for a vote.  These releases will be
    > > > > small and
    > > > > >     > > > testable enough that we can get feedback quickly and move
    > > from
    > > > > RC to
    > > > > >     > > > release in a short time.  If a release has too many issues
    > > > > then we may
    > > > > >     > > end
    > > > > >     > > > up deciding to skip that minor release in favor of the next
    > > > > one,
    > > > > >     > > hopefully
    > > > > >     > > > this does not happen very often if at all.
    > > > > >     > > >
    > > > > >     > > > Once a breaking change is introduced to master which will
    > > > > require a new
    > > > > >     > > > major release, we will create a new major release branch
    > > from
    > > > > master
    > > > > >     > > (which
    > > > > >     > > > will mean a new release manager).  We will then repeat the
    > > > > same process
    > > > > >     > > > with the new release branch, etc, etc.
    > > > > >     > > >
    > > > > >     > > > As for LTS we will provide support for the latest major
    > > > > release plus
    > > > > >     > the
    > > > > >     > > > one previous.  So, once we release 4.0 we will support 4.0
    > > and
    > > > > 3.1.
    > > > > >     > Any
    > > > > >     > > > security issues that arise will -- if present -- be
    > > applied to
    > > > > each of
    > > > > >     > > > these versions.  Once 4.1 is released we will support 4.1
    > > and
    > > > > 3.1, etc,
    > > > > >     > > > etc.
    > > > > >     > > >
    > > > > >     > > > Please let me know if you have any thoughts on this plan.
    > > If
    > > > > there are
    > > > > >     > > no
    > > > > >     > > > major objections I would like to try this with 4.x with the
    > > > > idea that
    > > > > >     > we
    > > > > >     > > > will adjust how we do things as necessary.
    > > > > >     > > >
    > > > > >     > > > We need to get better at releasing and something needs to
    > > > > change,
    > > > > >     > > hopefully
    > > > > >     > > > this can get us going in the right direction.
    > > > > >     > > >
    > > > > >     > > > Thanks,
    > > > > >     > > > Dave
    > > > > >     > > >
    > > > > >     > > >
    > > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
    > > > > cherry-picks to
    > > > > >     > our
    > > > > >     > > > release branch to control what goes into a minor release.
    > > > > Major
    > > > > >     > releases
    > > > > >     > > > will be cut from master as necessary - such as if there is
    > > a
    > > > > breaking
    > > > > >     > > > change that is introduced.
    > > > > >     > > >
    > > > > >     > >
    > > > > >     >
    > > > > >
    > > > > >
    > > > >
    > >

Re: [EXTERNAL] Re: Release Process

[Rawlin Peters <ra...@apache.org>]

Alright, I think that sounds fair. In general what I'm hearing is,
every 4-6 weeks:
1. fast-forward 4.x to the most recent "stable" commit of master
(maybe we automate this and fast-forward every day if some set of
tests pass, unless we explicitly know master is unstable for other
reasons)
2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
4.*.x that are necessary for the release (e.g. bugs that break the
data plane or prevent operators from being able to do their job or run
the control plane)

Which means the release manager needs to know/track:
- unstable commits along with their corresponding fixes
- incomplete features
- features we'd like to hold until the next release (due to current
release having enough already)
- any cherry-picks that were deemed necessary for a certain release
(so that they can either be carried forward into the next minor
release or, ideally, included as part of fast-forwarding 4.x to the
most recent stable commit on master)

I think this will work. We gain the ability to:
1. let master be "unstable" periodically
2. control the size of releases
3. make releases mirror master as closely as possible (because they
will just be a snapshot of the most recent version of master that was
considered stable -- unless absolutely necessary bug/security fixes
were cherry-picked in)

Does that all sound right, Dave?

- Rawlin







On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com> wrote:
>
> I think a third reason would be that we are getting very close to our
> release branch date, it's a large/complex feature, and we feel like the
> release already has enough in it.  In that case we could hold off on adding
> that change to the 4.x branch until after we cut the 4.x.x branch.  The
> benefit is that we can still merge that feature into master and get all of
> our testing (both internally and externally) running against it.   If you
> stick with just branching off master and this situation arises then we get
> in the spot where we just don't merge the PR at all until after the release
> branch is cut (we have two examples of this right now).  I personally think
> it's better to get this into master and let our automation run against it
> than it is to leave it out in a PR even though it is ready.
>
> As for the process, I was thinking the same thing except that you don't
> even need to cherry-pick.  In your scenario above you can just rebase
> master into 4.x once E was merged.  This avoids the headaches of having to
> cherry pick each commit while still giving us the benefit of having all of
> the changes on the release branch.  We can even decide to setup automation
> to rebase master into the release branch on an interval or on demand.  If
> we rebase in something that turns out to be broken we just roll back to the
> previous rebase (or a certain commit) until that is fixed and then rebase
> again.  If a feature gets merged to master that we aren't ready to put into
> the release, then we stop the automated rebase until we are ready to get it
> into the release -- at that point we should consider ONLY cherry-picking
> bug fixes for the remainder of that release cycle to make things easier to
> rebase again.
>
> I am sure there will be unintended consequences, and I am sure we will have
> challenges to work through, but I still think that it's worth trying.  We
> should fail fast and be willing to change course if necessary.
> I think if we can be consistent enough with our releases then maybe we can
> discuss branching from master again.
>
> Thanks,
> Dave
>
>
>
> On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
>
> > I guess my concern is really about what possible reasons we might have
> > to not cherry-pick a specific PR that had been merged into master
> > already. The only reasons I can think of to not cherry-pick something
> > would be:
> > 1. the PR was completely broken (e.g. breaks the build, tests don't
> > pass, doesn't compile, etc).
> > 2. the PR one one of a larger multi-PR "feature", and *we know*
> > releasing some parts early could cause problems (that knowing part is
> > crucial)
> >
> > We should always strive to catch issue #1 in our CI before it merges
> > into master. When things like that merge it can block people from
> > doing development starting at that point until it's fixed.
> > I can see why issue #2 might arise due to multiple different people
> > developing one larger feature, but we should still strive to break
> > larger features up into individual pieces that can reasonably stand on
> > their own if possible. Due to time constraints or other reasons I can
> > see that might be a little difficult.
> >
> > That said, I would still strive to make the 4.x release branch a
> > snapshot of master at some "known good point" in time. Let's say we
> > cut 4.x at commit A, then a broken commit B gets merged into master.
> > If it's completely obvious that it's a broken commit, I'd wait until
> > at least a commit C gets merged to fix it. Then I would cherry-pick B
> > and C into 4.x at the same time. If the fix is really like a commit E
> > with unrelated commits C and D in between, ideally I would still wait
> > to cherry-pick C and D until a "fix" commit E was merged (to fix
> > commit B), at which point I'd cherrypick commits B through E at the
> > same time, in order, to avoid any possible merge conflicts.
> >
> > I think to make that process easier for myself I would just
> > cherry-pick everything into the 4.x branch as it gets merged into
> > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
> > commits were not fixed/completed in time, I would reset the 4.x branch
> > to the last stable commit. This would mean that unstable
> > commits/incomplete features could potentially block things from going
> > into the release. I might count that as a positive because it would
> > incentivize us to improve our CI and do our best to also keep the
> > master branch as stable as possible.
> >
> > The end result would be the same as just continually jumping up the
> > master branch from known good, stable points in time to the next,
> > creating a minor release branch at each of those points.
> >
> > - Rawlin
> >
> >
> >
> > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
> > >
> > > Sorry, I have taken too long to respond.
> > >
> > > +1000000 on better CI and Testing, I think we all agree that needs to get
> > > better and is a different beast than this topic.
> > >
> > > As for the release branches, I am trying to follow a process similar to
> > > ATS, which is usually the model we try to follow with our project.
> > > While I agree that branching from master is the ideal solution, we tried
> > > that and it didn't work.  I feel like we are trying to agree to do the
> > same
> > > thing that we already tried and didn't work.  I think we should have a
> > > little freedom to commit things into master and, if they break things,
> > work
> > > on getting them fixed.  We also might commit PRs to master that are
> > partial
> > > implementations of a bigger feature.   If we are using a time based
> > > strategy to cut our releases from master then we are going to be
> > > continually worrying about making sure Master stays not broken.  I think
> > we
> > > should try to keep master deployable, but it's fine if it's not for short
> > > periods of time.
> > >
> > > I am not trying to suggest that we are "picky" about what goes into our
> > > release branch.  I think we should try to cherry-pick as much as we can
> > > into our release branch once it is ready to be cherry-picked.  I think we
> > > should be able to take the tip of our release branch and deploy it
> > whenever
> > > we want with some confidence since we know that features/bug-fixes that
> > > have been cherry-picked have already been reviewed and tested against
> > > master.  Yes, its more work, but it also give us the ability to have more
> > > confidence in what is in our release at any given time.  Again, we should
> > > be trying to cherry-pick in everything that makes sense to cherry-pick.
> > >
> > > I think the process that I would like to see looks like this:
> > > - Cut a 4.x branch from Master
> > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > branch
> > > - Tag the 4.0.0 release from the 4.0.x branch
> > > - New non-breaking changes and bug fixes that go into master are also
> > > Cherry Picked to 4.x
> > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
> > > are ready for a RC.
> > > - Rinse and repeat for all 4.x releases
> > > - Once there's a need for a major release we make a 5.x branch from
> > master
> > >
> > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > then
> > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > have
> > > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
> > > really that bad and B) it gives us more control.
> > >
> > > This should allow us to deploy much quicker and with more confidence than
> > > is possible today.   We can also choose to deploy from any point in time
> > > from the 4.x branch with relatively high confidence.
> > >
> > > I'll wrap this up by saying that I don't think there's a perfect
> > solution,
> > > but I think we can agree on something that is worth trying at least with
> > > the 4.x release life cycle.  Once we have some experience, we can figure
> > > out where it is not meeting our needs and figure out how to adjust as
> > > necessary.  I think we can all agree our current process is not working.
> > >
> > > Thanks,
> > > Dave
> > >
> > >
> > >
> > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > wrote:
> > >
> > > > +1 on more frequent releases
> > > > +1 on improving our CI. We need to get our CI to the point where we're
> > > > confident deploying changes to production if the CI passes.
> > > > +1 on "trunk-based" development as opposed to GitFlow
> > > > -1 on cherry-picking new features. This is risky, tedious, and
> > > > represents unnecessary effort that would be better spent elsewhere,
> > > > like improving our CI or adding more automated tests.
> > > > +1 on cutting minor release branches directly off master
> > > >
> > > > I think cutting minor release branches directly off master can put us
> > > > in a win-win scenario. Our releases become smaller and more frequent,
> > > > and we avoid the unnecessary risk and effort of cherry-picking new
> > > > features onto stale release branches.
> > > >
> > > > This is the process that I would propose:
> > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > > > 2. Develop master for 4-6 weeks.
> > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > > > 5. Whenever a major (show-stopping) bug or security issue is found,
> > > > cherry-pick the fix from master into the latest release branch (e.g.
> > > > 4.1.x), and create a 4.1.1 RC.
> > > >
> > > > As the 4.x release manager, I really don't want to get into the
> > > > business of selecting what should and should not go into a particular
> > > > release. Every 4-6 weeks, we should just cut a new minor release
> > > > branch off master and take everything that was contributed during that
> > > > time, as long as everything builds fine and all the tests pass. The
> > > > master branch, while it doesn't always have to be perfect, should at
> > > > least always be deployable. If a new feature is merged with bugs and
> > > > becomes a show-stopper, it should be either immediately remedied or
> > > > reverted on master while we work on a fix.
> > > >
> > > > - Rawlin
> > > >
> > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > > > <Mi...@comcast.com> wrote:
> > > > >
> > > > > In my opinion, in order to get to a cadence we are discussing we
> > need to
> > > > put a lot more work into the CI system. It has been failing
> > consistently,
> > > > doesn’t block PR approvals/merges and when it actually runs it does not
> > > > test anything besides build and license headers.
> > > > >
> > > > > In the past couple of weeks we have had (off the top of my head):
> > > > >
> > > > > * Unit tests broken
> > > > > * TO API tests broken
> > > > > * Golang vendor issues
> > > > > * TO Go build issues
> > > > >
> > > > > All were merged and broke master.
> > > > >
> > > > > Yes, having release branches sounds great and more aggressive cadence
> > > > will minimize the amount of risk but I think that comes with the need
> > to
> > > > improve on our automated validation and testing.
> > > > >
> > > > > We have proven that we have not kept master in a good state and
> > adding
> > > > more releases (more branches) will make it even harder to keep that
> > > > stability we are already fighting.
> > > > >
> > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> > wrote:
> > > > >
> > > > >     Yeah, I get it. No one company should be driving release
> > > > schedules/scope.
> > > > >
> > > > >     What I was really getting at is if ANY company has a recent
> > version
> > > > in a
> > > > >     test or prod-like enviro (meaning the version is being exercised
> > and
> > > > >     thoroughly tested), then maybe we consider simply cutting a
> > release
> > > > from
> > > > >     that version. For example, imagine company X had this version in
> > a
> > > > >     test/prod enviro:
> > > > >
> > > > >     Master-10287.7e62d07 (
> > > > >
> > > >
> > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > > > )
> > > > >
> > > > >     Then I would be in favor of cutting the 4.0 release from
> > > > >     Master-10287.7e62d07 as we know it is proven to work based on
> > > > company X's
> > > > >     actual use of it.
> > > > >
> > > > >     The truth is (imo), we don't have the bandwidth to manually
> > verify
> > > > releases
> > > > >     that we are not using/have not used nor do we have the necessary
> > > > automated
> > > > >     test coverage to verify these releases. So most of our releases
> > are
> > > > >     significantly untested.
> > > > >
> > > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> > > > wrote:
> > > > >
> > > > >     > I'm really not a fan of allowing Comcast to dictate the release
> > > > scope and
> > > > >     > schedule. If cherry-picking is too messy, then we can just cut
> > new
> > > > minor
> > > > >     > releases directly from master.
> > > > >     >
> > > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > mitchell852@gmail.com>
> > > > wrote:
> > > > >     >
> > > > >     > > I don't think it's as easy as cherry picking (backporting)
> > > > certain
> > > > >     > features
> > > > >     > > into a release branch. I could be wrong but I really don't
> > think
> > > > it is.
> > > > >     > >
> > > > >     > > So what I'm hearing is that 4.0 gets cut from master and we
> > go
> > > > through
> > > > >     > with
> > > > >     > > our normal process of testing, validating, etc. At the same
> > > > time, 4.1
> > > > >     > > branch is created from 4.0. Master moves on as normal. Let's
> > say
> > > > 25
> > > > >     > commits
> > > > >     > > come in to master during the next 6 weeks. During that 6
> > weeks,
> > > > we cherry
> > > > >     > > pick only the features that look interesting for 4.1? (the
> > rest
> > > > just stay
> > > > >     > > in master and will get picked up in 5.0?)
> > > > >     > >
> > > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
> > > > weeks apart
> > > > >     > and
> > > > >     > > have only "interesting" features so the releases are
> > > > incrementally very
> > > > >     > > small and much easier to test/validate, however, those
> > features
> > > > might
> > > > >     > > depend on other commits. At some point, cherry picking
> > becomes
> > > > impossible
> > > > >     > > as the foundation of the code base has shifted so
> > drastically.
> > > > Cherry
> > > > >     > > picking will become very messy and introduce a high level of
> > > > risk imo.
> > > > >     > >
> > > > >     > > Also, you'd have to manage the scope of these minor releases
> > > > which we are
> > > > >     > > not very good at because the designated release manager has a
> > > > full-time
> > > > >     > job
> > > > >     > > to attend to and now we have even more releases to
> > > > >     > > scope/test/validate/approve. I like the idea of faster
> > releases
> > > > but this
> > > > >     > > sounds like more work that nobody has the bandwidth for.
> > > > >     > >
> > > > >     > > Ok, so rather than shooting down an idea with no alternative
> > > > proposal, I
> > > > >     > > propose this. At Comcast, we are trying to get to the point
> > > > where we
> > > > >     > > release TC to prod from the head of master every few weeks.
> > What
> > > > if every
> > > > >     > > time Comcast does an internal upgrade, they note the commit
> > hash
> > > > and once
> > > > >     > > comcast is happy with the upgrade in their prod enviro, they
> > > > propose a
> > > > >     > > release pinned to that commit hash? With this approach you'd
> > end
> > > > up with:
> > > > >     > >
> > > > >     > > 1. fast releases (hopefully every few weeks)
> > > > >     > > 2. well tested releases as it's in a real prod enviro
> > > > >     > > 3. little/no validation work for anyone
> > > > >     > >
> > > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
> > > > front at the
> > > > >     > > edge of master and running it in a trusted enviro, they could
> > > > propose a
> > > > >     > > release as well.
> > > > >     > >
> > > > >     > > Jeremy
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > >
> > > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > > > david.neuman64@gmail.com>
> > > > >     > > wrote:
> > > > >     > >
> > > > >     > > > I have been thinking about how we can get better with our
> > > > release
> > > > >     > > cadence.
> > > > >     > > > I feel like we have slowed to a crawl and not been as good
> > as
> > > > we should
> > > > >     > > > about how we release.  Our last Major release was in March
> > and
> > > > we
> > > > >     > haven't
> > > > >     > > > had a real release since.   Moving forward I would like to
> > see
> > > > us get
> > > > >     > on
> > > > >     > > a
> > > > >     > > > more consistent cadence and provide smaller releases that
> > can
> > > > be more
> > > > >     > > > easily digested by the community.  It's my hope that we can
> > > > get to a
> > > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> > we are
> > > > >     > releasing
> > > > >     > > > in such a way that not all releases all required (e.g. if
> > you
> > > > are on
> > > > >     > 4.0
> > > > >     > > > you don't need 4.1, you can just install 4.2), and we are
> > more
> > > > >     > deliberate
> > > > >     > > > about what we put into a release.
> > > > >     > > >
> > > > >     > > > I think we can accomplish this by using our release
> > branches
> > > > better.
> > > > >     > For
> > > > >     > > > each major release we will cut a release branch from master
> > > > (e.g. 4.x)
> > > > >     > > and
> > > > >     > > > then we will use cherry-picking to add new features and bug
> > > > fixes to
> > > > >     > that
> > > > >     > > > release.  This means that if 4.0 has been released and you
> > > > want to get
> > > > >     > > your
> > > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
> > > > master and
> > > > >     > then
> > > > >     > > > cheery pick your squash merged commit to the 4.x branch
> > which
> > > > we will
> > > > >     > use
> > > > >     > > > to create the 4.1 release.  I think we should allow either
> > the
> > > > >     > > contributor
> > > > >     > > > or the committer (who is merging the PR) to suggest if a
> > > > feature goes
> > > > >     > > into
> > > > >     > > > the release branch, and if we disagree we can take it to
> > the
> > > > list.  If
> > > > >     > we
> > > > >     > > > decide that every PR to master also goes into 4.1 then so
> > be
> > > > it, but at
> > > > >     > > > least we are making a conscious decision of what goes into
> > the
> > > > next
> > > > >     > > > release.   This will allow us to not be so worried about
> > what
> > > > we are
> > > > >     > > > merging into master and how that will affect our next
> > release.
> > > > >     > According
> > > > >     > > > to our 6 week cadence we will cut a new release from the
> > > > current
> > > > >     > feature
> > > > >     > > > branch and put that up for a vote.  These releases will be
> > > > small and
> > > > >     > > > testable enough that we can get feedback quickly and move
> > from
> > > > RC to
> > > > >     > > > release in a short time.  If a release has too many issues
> > > > then we may
> > > > >     > > end
> > > > >     > > > up deciding to skip that minor release in favor of the next
> > > > one,
> > > > >     > > hopefully
> > > > >     > > > this does not happen very often if at all.
> > > > >     > > >
> > > > >     > > > Once a breaking change is introduced to master which will
> > > > require a new
> > > > >     > > > major release, we will create a new major release branch
> > from
> > > > master
> > > > >     > > (which
> > > > >     > > > will mean a new release manager).  We will then repeat the
> > > > same process
> > > > >     > > > with the new release branch, etc, etc.
> > > > >     > > >
> > > > >     > > > As for LTS we will provide support for the latest major
> > > > release plus
> > > > >     > the
> > > > >     > > > one previous.  So, once we release 4.0 we will support 4.0
> > and
> > > > 3.1.
> > > > >     > Any
> > > > >     > > > security issues that arise will -- if present -- be
> > applied to
> > > > each of
> > > > >     > > > these versions.  Once 4.1 is released we will support 4.1
> > and
> > > > 3.1, etc,
> > > > >     > > > etc.
> > > > >     > > >
> > > > >     > > > Please let me know if you have any thoughts on this plan.
> > If
> > > > there are
> > > > >     > > no
> > > > >     > > > major objections I would like to try this with 4.x with the
> > > > idea that
> > > > >     > we
> > > > >     > > > will adjust how we do things as necessary.
> > > > >     > > >
> > > > >     > > > We need to get better at releasing and something needs to
> > > > change,
> > > > >     > > hopefully
> > > > >     > > > this can get us going in the right direction.
> > > > >     > > >
> > > > >     > > > Thanks,
> > > > >     > > > Dave
> > > > >     > > >
> > > > >     > > >
> > > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > > > cherry-picks to
> > > > >     > our
> > > > >     > > > release branch to control what goes into a minor release.
> > > > Major
> > > > >     > releases
> > > > >     > > > will be cut from master as necessary - such as if there is
> > a
> > > > breaking
> > > > >     > > > change that is introduced.
> > > > >     > > >
> > > > >     > >
> > > > >     >
> > > > >
> > > > >
> > > >
> >

Re: [EXTERNAL] Re: Release Process

[David Neuman <da...@gmail.com>]

I think a third reason would be that we are getting very close to our
release branch date, it's a large/complex feature, and we feel like the
release already has enough in it.  In that case we could hold off on adding
that change to the 4.x branch until after we cut the 4.x.x branch.  The
benefit is that we can still merge that feature into master and get all of
our testing (both internally and externally) running against it.   If you
stick with just branching off master and this situation arises then we get
in the spot where we just don't merge the PR at all until after the release
branch is cut (we have two examples of this right now).  I personally think
it's better to get this into master and let our automation run against it
than it is to leave it out in a PR even though it is ready.

As for the process, I was thinking the same thing except that you don't
even need to cherry-pick.  In your scenario above you can just rebase
master into 4.x once E was merged.  This avoids the headaches of having to
cherry pick each commit while still giving us the benefit of having all of
the changes on the release branch.  We can even decide to setup automation
to rebase master into the release branch on an interval or on demand.  If
we rebase in something that turns out to be broken we just roll back to the
previous rebase (or a certain commit) until that is fixed and then rebase
again.  If a feature gets merged to master that we aren't ready to put into
the release, then we stop the automated rebase until we are ready to get it
into the release -- at that point we should consider ONLY cherry-picking
bug fixes for the remainder of that release cycle to make things easier to
rebase again.

I am sure there will be unintended consequences, and I am sure we will have
challenges to work through, but I still think that it's worth trying.  We
should fail fast and be willing to change course if necessary.
I think if we can be consistent enough with our releases then maybe we can
discuss branching from master again.

Thanks,
Dave



On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:

> I guess my concern is really about what possible reasons we might have
> to not cherry-pick a specific PR that had been merged into master
> already. The only reasons I can think of to not cherry-pick something
> would be:
> 1. the PR was completely broken (e.g. breaks the build, tests don't
> pass, doesn't compile, etc).
> 2. the PR one one of a larger multi-PR "feature", and *we know*
> releasing some parts early could cause problems (that knowing part is
> crucial)
>
> We should always strive to catch issue #1 in our CI before it merges
> into master. When things like that merge it can block people from
> doing development starting at that point until it's fixed.
> I can see why issue #2 might arise due to multiple different people
> developing one larger feature, but we should still strive to break
> larger features up into individual pieces that can reasonably stand on
> their own if possible. Due to time constraints or other reasons I can
> see that might be a little difficult.
>
> That said, I would still strive to make the 4.x release branch a
> snapshot of master at some "known good point" in time. Let's say we
> cut 4.x at commit A, then a broken commit B gets merged into master.
> If it's completely obvious that it's a broken commit, I'd wait until
> at least a commit C gets merged to fix it. Then I would cherry-pick B
> and C into 4.x at the same time. If the fix is really like a commit E
> with unrelated commits C and D in between, ideally I would still wait
> to cherry-pick C and D until a "fix" commit E was merged (to fix
> commit B), at which point I'd cherrypick commits B through E at the
> same time, in order, to avoid any possible merge conflicts.
>
> I think to make that process easier for myself I would just
> cherry-pick everything into the 4.x branch as it gets merged into
> master. Then, after 4-6 weeks of cherry-picking commits, if unstable
> commits were not fixed/completed in time, I would reset the 4.x branch
> to the last stable commit. This would mean that unstable
> commits/incomplete features could potentially block things from going
> into the release. I might count that as a positive because it would
> incentivize us to improve our CI and do our best to also keep the
> master branch as stable as possible.
>
> The end result would be the same as just continually jumping up the
> master branch from known good, stable points in time to the next,
> creating a minor release branch at each of those points.
>
> - Rawlin
>
>
>
> On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
> >
> > Sorry, I have taken too long to respond.
> >
> > +1000000 on better CI and Testing, I think we all agree that needs to get
> > better and is a different beast than this topic.
> >
> > As for the release branches, I am trying to follow a process similar to
> > ATS, which is usually the model we try to follow with our project.
> > While I agree that branching from master is the ideal solution, we tried
> > that and it didn't work.  I feel like we are trying to agree to do the
> same
> > thing that we already tried and didn't work.  I think we should have a
> > little freedom to commit things into master and, if they break things,
> work
> > on getting them fixed.  We also might commit PRs to master that are
> partial
> > implementations of a bigger feature.   If we are using a time based
> > strategy to cut our releases from master then we are going to be
> > continually worrying about making sure Master stays not broken.  I think
> we
> > should try to keep master deployable, but it's fine if it's not for short
> > periods of time.
> >
> > I am not trying to suggest that we are "picky" about what goes into our
> > release branch.  I think we should try to cherry-pick as much as we can
> > into our release branch once it is ready to be cherry-picked.  I think we
> > should be able to take the tip of our release branch and deploy it
> whenever
> > we want with some confidence since we know that features/bug-fixes that
> > have been cherry-picked have already been reviewed and tested against
> > master.  Yes, its more work, but it also give us the ability to have more
> > confidence in what is in our release at any given time.  Again, we should
> > be trying to cherry-pick in everything that makes sense to cherry-pick.
> >
> > I think the process that I would like to see looks like this:
> > - Cut a 4.x branch from Master
> > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> branch
> > - Tag the 4.0.0 release from the 4.0.x branch
> > - New non-breaking changes and bug fixes that go into master are also
> > Cherry Picked to 4.x
> > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
> > are ready for a RC.
> > - Rinse and repeat for all 4.x releases
> > - Once there's a need for a major release we make a 5.x branch from
> master
> >
> > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> then
> > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
> have
> > 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
> > really that bad and B) it gives us more control.
> >
> > This should allow us to deploy much quicker and with more confidence than
> > is possible today.   We can also choose to deploy from any point in time
> > from the 4.x branch with relatively high confidence.
> >
> > I'll wrap this up by saying that I don't think there's a perfect
> solution,
> > but I think we can agree on something that is worth trying at least with
> > the 4.x release life cycle.  Once we have some experience, we can figure
> > out where it is not meeting our needs and figure out how to adjust as
> > necessary.  I think we can all agree our current process is not working.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> wrote:
> >
> > > +1 on more frequent releases
> > > +1 on improving our CI. We need to get our CI to the point where we're
> > > confident deploying changes to production if the CI passes.
> > > +1 on "trunk-based" development as opposed to GitFlow
> > > -1 on cherry-picking new features. This is risky, tedious, and
> > > represents unnecessary effort that would be better spent elsewhere,
> > > like improving our CI or adding more automated tests.
> > > +1 on cutting minor release branches directly off master
> > >
> > > I think cutting minor release branches directly off master can put us
> > > in a win-win scenario. Our releases become smaller and more frequent,
> > > and we avoid the unnecessary risk and effort of cherry-picking new
> > > features onto stale release branches.
> > >
> > > This is the process that I would propose:
> > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > > 2. Develop master for 4-6 weeks.
> > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > > 5. Whenever a major (show-stopping) bug or security issue is found,
> > > cherry-pick the fix from master into the latest release branch (e.g.
> > > 4.1.x), and create a 4.1.1 RC.
> > >
> > > As the 4.x release manager, I really don't want to get into the
> > > business of selecting what should and should not go into a particular
> > > release. Every 4-6 weeks, we should just cut a new minor release
> > > branch off master and take everything that was contributed during that
> > > time, as long as everything builds fine and all the tests pass. The
> > > master branch, while it doesn't always have to be perfect, should at
> > > least always be deployable. If a new feature is merged with bugs and
> > > becomes a show-stopper, it should be either immediately remedied or
> > > reverted on master while we work on a fix.
> > >
> > > - Rawlin
> > >
> > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > > <Mi...@comcast.com> wrote:
> > > >
> > > > In my opinion, in order to get to a cadence we are discussing we
> need to
> > > put a lot more work into the CI system. It has been failing
> consistently,
> > > doesn’t block PR approvals/merges and when it actually runs it does not
> > > test anything besides build and license headers.
> > > >
> > > > In the past couple of weeks we have had (off the top of my head):
> > > >
> > > > * Unit tests broken
> > > > * TO API tests broken
> > > > * Golang vendor issues
> > > > * TO Go build issues
> > > >
> > > > All were merged and broke master.
> > > >
> > > > Yes, having release branches sounds great and more aggressive cadence
> > > will minimize the amount of risk but I think that comes with the need
> to
> > > improve on our automated validation and testing.
> > > >
> > > > We have proven that we have not kept master in a good state and
> adding
> > > more releases (more branches) will make it even harder to keep that
> > > stability we are already fighting.
> > > >
> > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> wrote:
> > > >
> > > >     Yeah, I get it. No one company should be driving release
> > > schedules/scope.
> > > >
> > > >     What I was really getting at is if ANY company has a recent
> version
> > > in a
> > > >     test or prod-like enviro (meaning the version is being exercised
> and
> > > >     thoroughly tested), then maybe we consider simply cutting a
> release
> > > from
> > > >     that version. For example, imagine company X had this version in
> a
> > > >     test/prod enviro:
> > > >
> > > >     Master-10287.7e62d07 (
> > > >
> > >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > > )
> > > >
> > > >     Then I would be in favor of cutting the 4.0 release from
> > > >     Master-10287.7e62d07 as we know it is proven to work based on
> > > company X's
> > > >     actual use of it.
> > > >
> > > >     The truth is (imo), we don't have the bandwidth to manually
> verify
> > > releases
> > > >     that we are not using/have not used nor do we have the necessary
> > > automated
> > > >     test coverage to verify these releases. So most of our releases
> are
> > > >     significantly untested.
> > > >
> > > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> > > wrote:
> > > >
> > > >     > I'm really not a fan of allowing Comcast to dictate the release
> > > scope and
> > > >     > schedule. If cherry-picking is too messy, then we can just cut
> new
> > > minor
> > > >     > releases directly from master.
> > > >     >
> > > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> mitchell852@gmail.com>
> > > wrote:
> > > >     >
> > > >     > > I don't think it's as easy as cherry picking (backporting)
> > > certain
> > > >     > features
> > > >     > > into a release branch. I could be wrong but I really don't
> think
> > > it is.
> > > >     > >
> > > >     > > So what I'm hearing is that 4.0 gets cut from master and we
> go
> > > through
> > > >     > with
> > > >     > > our normal process of testing, validating, etc. At the same
> > > time, 4.1
> > > >     > > branch is created from 4.0. Master moves on as normal. Let's
> say
> > > 25
> > > >     > commits
> > > >     > > come in to master during the next 6 weeks. During that 6
> weeks,
> > > we cherry
> > > >     > > pick only the features that look interesting for 4.1? (the
> rest
> > > just stay
> > > >     > > in master and will get picked up in 5.0?)
> > > >     > >
> > > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
> > > weeks apart
> > > >     > and
> > > >     > > have only "interesting" features so the releases are
> > > incrementally very
> > > >     > > small and much easier to test/validate, however, those
> features
> > > might
> > > >     > > depend on other commits. At some point, cherry picking
> becomes
> > > impossible
> > > >     > > as the foundation of the code base has shifted so
> drastically.
> > > Cherry
> > > >     > > picking will become very messy and introduce a high level of
> > > risk imo.
> > > >     > >
> > > >     > > Also, you'd have to manage the scope of these minor releases
> > > which we are
> > > >     > > not very good at because the designated release manager has a
> > > full-time
> > > >     > job
> > > >     > > to attend to and now we have even more releases to
> > > >     > > scope/test/validate/approve. I like the idea of faster
> releases
> > > but this
> > > >     > > sounds like more work that nobody has the bandwidth for.
> > > >     > >
> > > >     > > Ok, so rather than shooting down an idea with no alternative
> > > proposal, I
> > > >     > > propose this. At Comcast, we are trying to get to the point
> > > where we
> > > >     > > release TC to prod from the head of master every few weeks.
> What
> > > if every
> > > >     > > time Comcast does an internal upgrade, they note the commit
> hash
> > > and once
> > > >     > > comcast is happy with the upgrade in their prod enviro, they
> > > propose a
> > > >     > > release pinned to that commit hash? With this approach you'd
> end
> > > up with:
> > > >     > >
> > > >     > > 1. fast releases (hopefully every few weeks)
> > > >     > > 2. well tested releases as it's in a real prod enviro
> > > >     > > 3. little/no validation work for anyone
> > > >     > >
> > > >     > > Also, this doesn't have to be Comcast. If anyone else is out
> > > front at the
> > > >     > > edge of master and running it in a trusted enviro, they could
> > > propose a
> > > >     > > release as well.
> > > >     > >
> > > >     > > Jeremy
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > >
> > > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > > david.neuman64@gmail.com>
> > > >     > > wrote:
> > > >     > >
> > > >     > > > I have been thinking about how we can get better with our
> > > release
> > > >     > > cadence.
> > > >     > > > I feel like we have slowed to a crawl and not been as good
> as
> > > we should
> > > >     > > > about how we release.  Our last Major release was in March
> and
> > > we
> > > >     > haven't
> > > >     > > > had a real release since.   Moving forward I would like to
> see
> > > us get
> > > >     > on
> > > >     > > a
> > > >     > > > more consistent cadence and provide smaller releases that
> can
> > > be more
> > > >     > > > easily digested by the community.  It's my hope that we can
> > > get to a
> > > >     > > > cadence where we are doing releases every 4 to 6 weeks,
> we are
> > > >     > releasing
> > > >     > > > in such a way that not all releases all required (e.g. if
> you
> > > are on
> > > >     > 4.0
> > > >     > > > you don't need 4.1, you can just install 4.2), and we are
> more
> > > >     > deliberate
> > > >     > > > about what we put into a release.
> > > >     > > >
> > > >     > > > I think we can accomplish this by using our release
> branches
> > > better.
> > > >     > For
> > > >     > > > each major release we will cut a release branch from master
> > > (e.g. 4.x)
> > > >     > > and
> > > >     > > > then we will use cherry-picking to add new features and bug
> > > fixes to
> > > >     > that
> > > >     > > > release.  This means that if 4.0 has been released and you
> > > want to get
> > > >     > > your
> > > >     > > > feature/bug fix in 4.1, you will first submit your PR to
> > > master and
> > > >     > then
> > > >     > > > cheery pick your squash merged commit to the 4.x branch
> which
> > > we will
> > > >     > use
> > > >     > > > to create the 4.1 release.  I think we should allow either
> the
> > > >     > > contributor
> > > >     > > > or the committer (who is merging the PR) to suggest if a
> > > feature goes
> > > >     > > into
> > > >     > > > the release branch, and if we disagree we can take it to
> the
> > > list.  If
> > > >     > we
> > > >     > > > decide that every PR to master also goes into 4.1 then so
> be
> > > it, but at
> > > >     > > > least we are making a conscious decision of what goes into
> the
> > > next
> > > >     > > > release.   This will allow us to not be so worried about
> what
> > > we are
> > > >     > > > merging into master and how that will affect our next
> release.
> > > >     > According
> > > >     > > > to our 6 week cadence we will cut a new release from the
> > > current
> > > >     > feature
> > > >     > > > branch and put that up for a vote.  These releases will be
> > > small and
> > > >     > > > testable enough that we can get feedback quickly and move
> from
> > > RC to
> > > >     > > > release in a short time.  If a release has too many issues
> > > then we may
> > > >     > > end
> > > >     > > > up deciding to skip that minor release in favor of the next
> > > one,
> > > >     > > hopefully
> > > >     > > > this does not happen very often if at all.
> > > >     > > >
> > > >     > > > Once a breaking change is introduced to master which will
> > > require a new
> > > >     > > > major release, we will create a new major release branch
> from
> > > master
> > > >     > > (which
> > > >     > > > will mean a new release manager).  We will then repeat the
> > > same process
> > > >     > > > with the new release branch, etc, etc.
> > > >     > > >
> > > >     > > > As for LTS we will provide support for the latest major
> > > release plus
> > > >     > the
> > > >     > > > one previous.  So, once we release 4.0 we will support 4.0
> and
> > > 3.1.
> > > >     > Any
> > > >     > > > security issues that arise will -- if present -- be
> applied to
> > > each of
> > > >     > > > these versions.  Once 4.1 is released we will support 4.1
> and
> > > 3.1, etc,
> > > >     > > > etc.
> > > >     > > >
> > > >     > > > Please let me know if you have any thoughts on this plan.
> If
> > > there are
> > > >     > > no
> > > >     > > > major objections I would like to try this with 4.x with the
> > > idea that
> > > >     > we
> > > >     > > > will adjust how we do things as necessary.
> > > >     > > >
> > > >     > > > We need to get better at releasing and something needs to
> > > change,
> > > >     > > hopefully
> > > >     > > > this can get us going in the right direction.
> > > >     > > >
> > > >     > > > Thanks,
> > > >     > > > Dave
> > > >     > > >
> > > >     > > >
> > > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > > cherry-picks to
> > > >     > our
> > > >     > > > release branch to control what goes into a minor release.
> > > Major
> > > >     > releases
> > > >     > > > will be cut from master as necessary - such as if there is
> a
> > > breaking
> > > >     > > > change that is introduced.
> > > >     > > >
> > > >     > >
> > > >     >
> > > >
> > > >
> > >
>

Re: [EXTERNAL] Re: Release Process

[Rawlin Peters <ra...@apache.org>]

I guess my concern is really about what possible reasons we might have
to not cherry-pick a specific PR that had been merged into master
already. The only reasons I can think of to not cherry-pick something
would be:
1. the PR was completely broken (e.g. breaks the build, tests don't
pass, doesn't compile, etc).
2. the PR one one of a larger multi-PR "feature", and *we know*
releasing some parts early could cause problems (that knowing part is
crucial)

We should always strive to catch issue #1 in our CI before it merges
into master. When things like that merge it can block people from
doing development starting at that point until it's fixed.
I can see why issue #2 might arise due to multiple different people
developing one larger feature, but we should still strive to break
larger features up into individual pieces that can reasonably stand on
their own if possible. Due to time constraints or other reasons I can
see that might be a little difficult.

That said, I would still strive to make the 4.x release branch a
snapshot of master at some "known good point" in time. Let's say we
cut 4.x at commit A, then a broken commit B gets merged into master.
If it's completely obvious that it's a broken commit, I'd wait until
at least a commit C gets merged to fix it. Then I would cherry-pick B
and C into 4.x at the same time. If the fix is really like a commit E
with unrelated commits C and D in between, ideally I would still wait
to cherry-pick C and D until a "fix" commit E was merged (to fix
commit B), at which point I'd cherrypick commits B through E at the
same time, in order, to avoid any possible merge conflicts.

I think to make that process easier for myself I would just
cherry-pick everything into the 4.x branch as it gets merged into
master. Then, after 4-6 weeks of cherry-picking commits, if unstable
commits were not fixed/completed in time, I would reset the 4.x branch
to the last stable commit. This would mean that unstable
commits/incomplete features could potentially block things from going
into the release. I might count that as a positive because it would
incentivize us to improve our CI and do our best to also keep the
master branch as stable as possible.

The end result would be the same as just continually jumping up the
master branch from known good, stable points in time to the next,
creating a minor release branch at each of those points.

- Rawlin



On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
>
> Sorry, I have taken too long to respond.
>
> +1000000 on better CI and Testing, I think we all agree that needs to get
> better and is a different beast than this topic.
>
> As for the release branches, I am trying to follow a process similar to
> ATS, which is usually the model we try to follow with our project.
> While I agree that branching from master is the ideal solution, we tried
> that and it didn't work.  I feel like we are trying to agree to do the same
> thing that we already tried and didn't work.  I think we should have a
> little freedom to commit things into master and, if they break things, work
> on getting them fixed.  We also might commit PRs to master that are partial
> implementations of a bigger feature.   If we are using a time based
> strategy to cut our releases from master then we are going to be
> continually worrying about making sure Master stays not broken.  I think we
> should try to keep master deployable, but it's fine if it's not for short
> periods of time.
>
> I am not trying to suggest that we are "picky" about what goes into our
> release branch.  I think we should try to cherry-pick as much as we can
> into our release branch once it is ready to be cherry-picked.  I think we
> should be able to take the tip of our release branch and deploy it whenever
> we want with some confidence since we know that features/bug-fixes that
> have been cherry-picked have already been reviewed and tested against
> master.  Yes, its more work, but it also give us the ability to have more
> confidence in what is in our release at any given time.  Again, we should
> be trying to cherry-pick in everything that makes sense to cherry-pick.
>
> I think the process that I would like to see looks like this:
> - Cut a 4.x branch from Master
> - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x branch
> - Tag the 4.0.0 release from the 4.0.x branch
> - New non-breaking changes and bug fixes that go into master are also
> Cherry Picked to 4.x
> - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
> are ready for a RC.
> - Rinse and repeat for all 4.x releases
> - Once there's a need for a major release we make a 5.x branch from master
>
> If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, then
> 4.0.x which we use to create the new 4.0.1 release (assuming we don't have
> 4.1 yet).  Yes, that is a more work than today, but I think A) it's not
> really that bad and B) it gives us more control.
>
> This should allow us to deploy much quicker and with more confidence than
> is possible today.   We can also choose to deploy from any point in time
> from the 4.x branch with relatively high confidence.
>
> I'll wrap this up by saying that I don't think there's a perfect solution,
> but I think we can agree on something that is worth trying at least with
> the 4.x release life cycle.  Once we have some experience, we can figure
> out where it is not meeting our needs and figure out how to adjust as
> necessary.  I think we can all agree our current process is not working.
>
> Thanks,
> Dave
>
>
>
> On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org> wrote:
>
> > +1 on more frequent releases
> > +1 on improving our CI. We need to get our CI to the point where we're
> > confident deploying changes to production if the CI passes.
> > +1 on "trunk-based" development as opposed to GitFlow
> > -1 on cherry-picking new features. This is risky, tedious, and
> > represents unnecessary effort that would be better spent elsewhere,
> > like improving our CI or adding more automated tests.
> > +1 on cutting minor release branches directly off master
> >
> > I think cutting minor release branches directly off master can put us
> > in a win-win scenario. Our releases become smaller and more frequent,
> > and we avoid the unnecessary risk and effort of cherry-picking new
> > features onto stale release branches.
> >
> > This is the process that I would propose:
> > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > 2. Develop master for 4-6 weeks.
> > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > 5. Whenever a major (show-stopping) bug or security issue is found,
> > cherry-pick the fix from master into the latest release branch (e.g.
> > 4.1.x), and create a 4.1.1 RC.
> >
> > As the 4.x release manager, I really don't want to get into the
> > business of selecting what should and should not go into a particular
> > release. Every 4-6 weeks, we should just cut a new minor release
> > branch off master and take everything that was contributed during that
> > time, as long as everything builds fine and all the tests pass. The
> > master branch, while it doesn't always have to be perfect, should at
> > least always be deployable. If a new feature is merged with bugs and
> > becomes a show-stopper, it should be either immediately remedied or
> > reverted on master while we work on a fix.
> >
> > - Rawlin
> >
> > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > <Mi...@comcast.com> wrote:
> > >
> > > In my opinion, in order to get to a cadence we are discussing we need to
> > put a lot more work into the CI system. It has been failing consistently,
> > doesn’t block PR approvals/merges and when it actually runs it does not
> > test anything besides build and license headers.
> > >
> > > In the past couple of weeks we have had (off the top of my head):
> > >
> > > * Unit tests broken
> > > * TO API tests broken
> > > * Golang vendor issues
> > > * TO Go build issues
> > >
> > > All were merged and broke master.
> > >
> > > Yes, having release branches sounds great and more aggressive cadence
> > will minimize the amount of risk but I think that comes with the need to
> > improve on our automated validation and testing.
> > >
> > > We have proven that we have not kept master in a good state and adding
> > more releases (more branches) will make it even harder to keep that
> > stability we are already fighting.
> > >
> > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
> > >
> > >     Yeah, I get it. No one company should be driving release
> > schedules/scope.
> > >
> > >     What I was really getting at is if ANY company has a recent version
> > in a
> > >     test or prod-like enviro (meaning the version is being exercised and
> > >     thoroughly tested), then maybe we consider simply cutting a release
> > from
> > >     that version. For example, imagine company X had this version in a
> > >     test/prod enviro:
> > >
> > >     Master-10287.7e62d07 (
> > >
> > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > )
> > >
> > >     Then I would be in favor of cutting the 4.0 release from
> > >     Master-10287.7e62d07 as we know it is proven to work based on
> > company X's
> > >     actual use of it.
> > >
> > >     The truth is (imo), we don't have the bandwidth to manually verify
> > releases
> > >     that we are not using/have not used nor do we have the necessary
> > automated
> > >     test coverage to verify these releases. So most of our releases are
> > >     significantly untested.
> > >
> > >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> > wrote:
> > >
> > >     > I'm really not a fan of allowing Comcast to dictate the release
> > scope and
> > >     > schedule. If cherry-picking is too messy, then we can just cut new
> > minor
> > >     > releases directly from master.
> > >     >
> > >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> > >     >
> > >     > > I don't think it's as easy as cherry picking (backporting)
> > certain
> > >     > features
> > >     > > into a release branch. I could be wrong but I really don't think
> > it is.
> > >     > >
> > >     > > So what I'm hearing is that 4.0 gets cut from master and we go
> > through
> > >     > with
> > >     > > our normal process of testing, validating, etc. At the same
> > time, 4.1
> > >     > > branch is created from 4.0. Master moves on as normal. Let's say
> > 25
> > >     > commits
> > >     > > come in to master during the next 6 weeks. During that 6 weeks,
> > we cherry
> > >     > > pick only the features that look interesting for 4.1? (the rest
> > just stay
> > >     > > in master and will get picked up in 5.0?)
> > >     > >
> > >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
> > weeks apart
> > >     > and
> > >     > > have only "interesting" features so the releases are
> > incrementally very
> > >     > > small and much easier to test/validate, however, those features
> > might
> > >     > > depend on other commits. At some point, cherry picking becomes
> > impossible
> > >     > > as the foundation of the code base has shifted so drastically.
> > Cherry
> > >     > > picking will become very messy and introduce a high level of
> > risk imo.
> > >     > >
> > >     > > Also, you'd have to manage the scope of these minor releases
> > which we are
> > >     > > not very good at because the designated release manager has a
> > full-time
> > >     > job
> > >     > > to attend to and now we have even more releases to
> > >     > > scope/test/validate/approve. I like the idea of faster releases
> > but this
> > >     > > sounds like more work that nobody has the bandwidth for.
> > >     > >
> > >     > > Ok, so rather than shooting down an idea with no alternative
> > proposal, I
> > >     > > propose this. At Comcast, we are trying to get to the point
> > where we
> > >     > > release TC to prod from the head of master every few weeks. What
> > if every
> > >     > > time Comcast does an internal upgrade, they note the commit hash
> > and once
> > >     > > comcast is happy with the upgrade in their prod enviro, they
> > propose a
> > >     > > release pinned to that commit hash? With this approach you'd end
> > up with:
> > >     > >
> > >     > > 1. fast releases (hopefully every few weeks)
> > >     > > 2. well tested releases as it's in a real prod enviro
> > >     > > 3. little/no validation work for anyone
> > >     > >
> > >     > > Also, this doesn't have to be Comcast. If anyone else is out
> > front at the
> > >     > > edge of master and running it in a trusted enviro, they could
> > propose a
> > >     > > release as well.
> > >     > >
> > >     > > Jeremy
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > >
> > >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > david.neuman64@gmail.com>
> > >     > > wrote:
> > >     > >
> > >     > > > I have been thinking about how we can get better with our
> > release
> > >     > > cadence.
> > >     > > > I feel like we have slowed to a crawl and not been as good as
> > we should
> > >     > > > about how we release.  Our last Major release was in March and
> > we
> > >     > haven't
> > >     > > > had a real release since.   Moving forward I would like to see
> > us get
> > >     > on
> > >     > > a
> > >     > > > more consistent cadence and provide smaller releases that can
> > be more
> > >     > > > easily digested by the community.  It's my hope that we can
> > get to a
> > >     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
> > >     > releasing
> > >     > > > in such a way that not all releases all required (e.g. if you
> > are on
> > >     > 4.0
> > >     > > > you don't need 4.1, you can just install 4.2), and we are more
> > >     > deliberate
> > >     > > > about what we put into a release.
> > >     > > >
> > >     > > > I think we can accomplish this by using our release branches
> > better.
> > >     > For
> > >     > > > each major release we will cut a release branch from master
> > (e.g. 4.x)
> > >     > > and
> > >     > > > then we will use cherry-picking to add new features and bug
> > fixes to
> > >     > that
> > >     > > > release.  This means that if 4.0 has been released and you
> > want to get
> > >     > > your
> > >     > > > feature/bug fix in 4.1, you will first submit your PR to
> > master and
> > >     > then
> > >     > > > cheery pick your squash merged commit to the 4.x branch which
> > we will
> > >     > use
> > >     > > > to create the 4.1 release.  I think we should allow either the
> > >     > > contributor
> > >     > > > or the committer (who is merging the PR) to suggest if a
> > feature goes
> > >     > > into
> > >     > > > the release branch, and if we disagree we can take it to the
> > list.  If
> > >     > we
> > >     > > > decide that every PR to master also goes into 4.1 then so be
> > it, but at
> > >     > > > least we are making a conscious decision of what goes into the
> > next
> > >     > > > release.   This will allow us to not be so worried about what
> > we are
> > >     > > > merging into master and how that will affect our next release.
> > >     > According
> > >     > > > to our 6 week cadence we will cut a new release from the
> > current
> > >     > feature
> > >     > > > branch and put that up for a vote.  These releases will be
> > small and
> > >     > > > testable enough that we can get feedback quickly and move from
> > RC to
> > >     > > > release in a short time.  If a release has too many issues
> > then we may
> > >     > > end
> > >     > > > up deciding to skip that minor release in favor of the next
> > one,
> > >     > > hopefully
> > >     > > > this does not happen very often if at all.
> > >     > > >
> > >     > > > Once a breaking change is introduced to master which will
> > require a new
> > >     > > > major release, we will create a new major release branch from
> > master
> > >     > > (which
> > >     > > > will mean a new release manager).  We will then repeat the
> > same process
> > >     > > > with the new release branch, etc, etc.
> > >     > > >
> > >     > > > As for LTS we will provide support for the latest major
> > release plus
> > >     > the
> > >     > > > one previous.  So, once we release 4.0 we will support 4.0 and
> > 3.1.
> > >     > Any
> > >     > > > security issues that arise will -- if present -- be applied to
> > each of
> > >     > > > these versions.  Once 4.1 is released we will support 4.1 and
> > 3.1, etc,
> > >     > > > etc.
> > >     > > >
> > >     > > > Please let me know if you have any thoughts on this plan.  If
> > there are
> > >     > > no
> > >     > > > major objections I would like to try this with 4.x with the
> > idea that
> > >     > we
> > >     > > > will adjust how we do things as necessary.
> > >     > > >
> > >     > > > We need to get better at releasing and something needs to
> > change,
> > >     > > hopefully
> > >     > > > this can get us going in the right direction.
> > >     > > >
> > >     > > > Thanks,
> > >     > > > Dave
> > >     > > >
> > >     > > >
> > >     > > > TL;DR -  I am proposing a 6 week release cycle using
> > cherry-picks to
> > >     > our
> > >     > > > release branch to control what goes into a minor release.
> > Major
> > >     > releases
> > >     > > > will be cut from master as necessary - such as if there is a
> > breaking
> > >     > > > change that is introduced.
> > >     > > >
> > >     > >
> > >     >
> > >
> > >
> >

Re: [EXTERNAL] Re: Release Process

[Dave Neuman <ne...@apache.org>]

Sorry, I have taken too long to respond.

+1000000 on better CI and Testing, I think we all agree that needs to get
better and is a different beast than this topic.

As for the release branches, I am trying to follow a process similar to
ATS, which is usually the model we try to follow with our project.
While I agree that branching from master is the ideal solution, we tried
that and it didn't work.  I feel like we are trying to agree to do the same
thing that we already tried and didn't work.  I think we should have a
little freedom to commit things into master and, if they break things, work
on getting them fixed.  We also might commit PRs to master that are partial
implementations of a bigger feature.   If we are using a time based
strategy to cut our releases from master then we are going to be
continually worrying about making sure Master stays not broken.  I think we
should try to keep master deployable, but it's fine if it's not for short
periods of time.

I am not trying to suggest that we are "picky" about what goes into our
release branch.  I think we should try to cherry-pick as much as we can
into our release branch once it is ready to be cherry-picked.  I think we
should be able to take the tip of our release branch and deploy it whenever
we want with some confidence since we know that features/bug-fixes that
have been cherry-picked have already been reviewed and tested against
master.  Yes, its more work, but it also give us the ability to have more
confidence in what is in our release at any given time.  Again, we should
be trying to cherry-pick in everything that makes sense to cherry-pick.

I think the process that I would like to see looks like this:
- Cut a 4.x branch from Master
- Once we think we are ready for a RC cut a 4.0.x branch from the 4.x branch
- Tag the 4.0.0 release from the 4.0.x branch
- New non-breaking changes and bug fixes that go into master are also
Cherry Picked to 4.x
- After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when we
are ready for a RC.
- Rinse and repeat for all 4.x releases
- Once there's a need for a major release we make a 5.x branch from master

If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, then
4.0.x which we use to create the new 4.0.1 release (assuming we don't have
4.1 yet).  Yes, that is a more work than today, but I think A) it's not
really that bad and B) it gives us more control.

This should allow us to deploy much quicker and with more confidence than
is possible today.   We can also choose to deploy from any point in time
from the 4.x branch with relatively high confidence.

I'll wrap this up by saying that I don't think there's a perfect solution,
but I think we can agree on something that is worth trying at least with
the 4.x release life cycle.  Once we have some experience, we can figure
out where it is not meeting our needs and figure out how to adjust as
necessary.  I think we can all agree our current process is not working.

Thanks,
Dave



On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org> wrote:

> +1 on more frequent releases
> +1 on improving our CI. We need to get our CI to the point where we're
> confident deploying changes to production if the CI passes.
> +1 on "trunk-based" development as opposed to GitFlow
> -1 on cherry-picking new features. This is risky, tedious, and
> represents unnecessary effort that would be better spent elsewhere,
> like improving our CI or adding more automated tests.
> +1 on cutting minor release branches directly off master
>
> I think cutting minor release branches directly off master can put us
> in a win-win scenario. Our releases become smaller and more frequent,
> and we avoid the unnecessary risk and effort of cherry-picking new
> features onto stale release branches.
>
> This is the process that I would propose:
> 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> 2. Develop master for 4-6 weeks.
> 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> 4. Repeat steps 2 and 3 for new minor releases as necessary.
> 5. Whenever a major (show-stopping) bug or security issue is found,
> cherry-pick the fix from master into the latest release branch (e.g.
> 4.1.x), and create a 4.1.1 RC.
>
> As the 4.x release manager, I really don't want to get into the
> business of selecting what should and should not go into a particular
> release. Every 4-6 weeks, we should just cut a new minor release
> branch off master and take everything that was contributed during that
> time, as long as everything builds fine and all the tests pass. The
> master branch, while it doesn't always have to be perfect, should at
> least always be deployable. If a new feature is merged with bugs and
> becomes a show-stopper, it should be either immediately remedied or
> reverted on master while we work on a fix.
>
> - Rawlin
>
> On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> <Mi...@comcast.com> wrote:
> >
> > In my opinion, in order to get to a cadence we are discussing we need to
> put a lot more work into the CI system. It has been failing consistently,
> doesn’t block PR approvals/merges and when it actually runs it does not
> test anything besides build and license headers.
> >
> > In the past couple of weeks we have had (off the top of my head):
> >
> > * Unit tests broken
> > * TO API tests broken
> > * Golang vendor issues
> > * TO Go build issues
> >
> > All were merged and broke master.
> >
> > Yes, having release branches sounds great and more aggressive cadence
> will minimize the amount of risk but I think that comes with the need to
> improve on our automated validation and testing.
> >
> > We have proven that we have not kept master in a good state and adding
> more releases (more branches) will make it even harder to keep that
> stability we are already fighting.
> >
> > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
> >
> >     Yeah, I get it. No one company should be driving release
> schedules/scope.
> >
> >     What I was really getting at is if ANY company has a recent version
> in a
> >     test or prod-like enviro (meaning the version is being exercised and
> >     thoroughly tested), then maybe we consider simply cutting a release
> from
> >     that version. For example, imagine company X had this version in a
> >     test/prod enviro:
> >
> >     Master-10287.7e62d07 (
> >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> )
> >
> >     Then I would be in favor of cutting the 4.0 release from
> >     Master-10287.7e62d07 as we know it is proven to work based on
> company X's
> >     actual use of it.
> >
> >     The truth is (imo), we don't have the bandwidth to manually verify
> releases
> >     that we are not using/have not used nor do we have the necessary
> automated
> >     test coverage to verify these releases. So most of our releases are
> >     significantly untested.
> >
> >     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com>
> wrote:
> >
> >     > I'm really not a fan of allowing Comcast to dictate the release
> scope and
> >     > schedule. If cherry-picking is too messy, then we can just cut new
> minor
> >     > releases directly from master.
> >     >
> >     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com>
> wrote:
> >     >
> >     > > I don't think it's as easy as cherry picking (backporting)
> certain
> >     > features
> >     > > into a release branch. I could be wrong but I really don't think
> it is.
> >     > >
> >     > > So what I'm hearing is that 4.0 gets cut from master and we go
> through
> >     > with
> >     > > our normal process of testing, validating, etc. At the same
> time, 4.1
> >     > > branch is created from 4.0. Master moves on as normal. Let's say
> 25
> >     > commits
> >     > > come in to master during the next 6 weeks. During that 6 weeks,
> we cherry
> >     > > pick only the features that look interesting for 4.1? (the rest
> just stay
> >     > > in master and will get picked up in 5.0?)
> >     > >
> >     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6
> weeks apart
> >     > and
> >     > > have only "interesting" features so the releases are
> incrementally very
> >     > > small and much easier to test/validate, however, those features
> might
> >     > > depend on other commits. At some point, cherry picking becomes
> impossible
> >     > > as the foundation of the code base has shifted so drastically.
> Cherry
> >     > > picking will become very messy and introduce a high level of
> risk imo.
> >     > >
> >     > > Also, you'd have to manage the scope of these minor releases
> which we are
> >     > > not very good at because the designated release manager has a
> full-time
> >     > job
> >     > > to attend to and now we have even more releases to
> >     > > scope/test/validate/approve. I like the idea of faster releases
> but this
> >     > > sounds like more work that nobody has the bandwidth for.
> >     > >
> >     > > Ok, so rather than shooting down an idea with no alternative
> proposal, I
> >     > > propose this. At Comcast, we are trying to get to the point
> where we
> >     > > release TC to prod from the head of master every few weeks. What
> if every
> >     > > time Comcast does an internal upgrade, they note the commit hash
> and once
> >     > > comcast is happy with the upgrade in their prod enviro, they
> propose a
> >     > > release pinned to that commit hash? With this approach you'd end
> up with:
> >     > >
> >     > > 1. fast releases (hopefully every few weeks)
> >     > > 2. well tested releases as it's in a real prod enviro
> >     > > 3. little/no validation work for anyone
> >     > >
> >     > > Also, this doesn't have to be Comcast. If anyone else is out
> front at the
> >     > > edge of master and running it in a trusted enviro, they could
> propose a
> >     > > release as well.
> >     > >
> >     > > Jeremy
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > >
> >     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> david.neuman64@gmail.com>
> >     > > wrote:
> >     > >
> >     > > > I have been thinking about how we can get better with our
> release
> >     > > cadence.
> >     > > > I feel like we have slowed to a crawl and not been as good as
> we should
> >     > > > about how we release.  Our last Major release was in March and
> we
> >     > haven't
> >     > > > had a real release since.   Moving forward I would like to see
> us get
> >     > on
> >     > > a
> >     > > > more consistent cadence and provide smaller releases that can
> be more
> >     > > > easily digested by the community.  It's my hope that we can
> get to a
> >     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
> >     > releasing
> >     > > > in such a way that not all releases all required (e.g. if you
> are on
> >     > 4.0
> >     > > > you don't need 4.1, you can just install 4.2), and we are more
> >     > deliberate
> >     > > > about what we put into a release.
> >     > > >
> >     > > > I think we can accomplish this by using our release branches
> better.
> >     > For
> >     > > > each major release we will cut a release branch from master
> (e.g. 4.x)
> >     > > and
> >     > > > then we will use cherry-picking to add new features and bug
> fixes to
> >     > that
> >     > > > release.  This means that if 4.0 has been released and you
> want to get
> >     > > your
> >     > > > feature/bug fix in 4.1, you will first submit your PR to
> master and
> >     > then
> >     > > > cheery pick your squash merged commit to the 4.x branch which
> we will
> >     > use
> >     > > > to create the 4.1 release.  I think we should allow either the
> >     > > contributor
> >     > > > or the committer (who is merging the PR) to suggest if a
> feature goes
> >     > > into
> >     > > > the release branch, and if we disagree we can take it to the
> list.  If
> >     > we
> >     > > > decide that every PR to master also goes into 4.1 then so be
> it, but at
> >     > > > least we are making a conscious decision of what goes into the
> next
> >     > > > release.   This will allow us to not be so worried about what
> we are
> >     > > > merging into master and how that will affect our next release.
> >     > According
> >     > > > to our 6 week cadence we will cut a new release from the
> current
> >     > feature
> >     > > > branch and put that up for a vote.  These releases will be
> small and
> >     > > > testable enough that we can get feedback quickly and move from
> RC to
> >     > > > release in a short time.  If a release has too many issues
> then we may
> >     > > end
> >     > > > up deciding to skip that minor release in favor of the next
> one,
> >     > > hopefully
> >     > > > this does not happen very often if at all.
> >     > > >
> >     > > > Once a breaking change is introduced to master which will
> require a new
> >     > > > major release, we will create a new major release branch from
> master
> >     > > (which
> >     > > > will mean a new release manager).  We will then repeat the
> same process
> >     > > > with the new release branch, etc, etc.
> >     > > >
> >     > > > As for LTS we will provide support for the latest major
> release plus
> >     > the
> >     > > > one previous.  So, once we release 4.0 we will support 4.0 and
> 3.1.
> >     > Any
> >     > > > security issues that arise will -- if present -- be applied to
> each of
> >     > > > these versions.  Once 4.1 is released we will support 4.1 and
> 3.1, etc,
> >     > > > etc.
> >     > > >
> >     > > > Please let me know if you have any thoughts on this plan.  If
> there are
> >     > > no
> >     > > > major objections I would like to try this with 4.x with the
> idea that
> >     > we
> >     > > > will adjust how we do things as necessary.
> >     > > >
> >     > > > We need to get better at releasing and something needs to
> change,
> >     > > hopefully
> >     > > > this can get us going in the right direction.
> >     > > >
> >     > > > Thanks,
> >     > > > Dave
> >     > > >
> >     > > >
> >     > > > TL;DR -  I am proposing a 6 week release cycle using
> cherry-picks to
> >     > our
> >     > > > release branch to control what goes into a minor release.
> Major
> >     > releases
> >     > > > will be cut from master as necessary - such as if there is a
> breaking
> >     > > > change that is introduced.
> >     > > >
> >     > >
> >     >
> >
> >
>

Re: [EXTERNAL] Re: Release Process

[Rawlin Peters <ra...@apache.org>]

+1 on more frequent releases
+1 on improving our CI. We need to get our CI to the point where we're
confident deploying changes to production if the CI passes.
+1 on "trunk-based" development as opposed to GitFlow
-1 on cherry-picking new features. This is risky, tedious, and
represents unnecessary effort that would be better spent elsewhere,
like improving our CI or adding more automated tests.
+1 on cutting minor release branches directly off master

I think cutting minor release branches directly off master can put us
in a win-win scenario. Our releases become smaller and more frequent,
and we avoid the unnecessary risk and effort of cherry-picking new
features onto stale release branches.

This is the process that I would propose:
1. Cut a 4.0.x branch off master which becomes the 4.0 RC
2. Develop master for 4-6 weeks.
3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
4. Repeat steps 2 and 3 for new minor releases as necessary.
5. Whenever a major (show-stopping) bug or security issue is found,
cherry-pick the fix from master into the latest release branch (e.g.
4.1.x), and create a 4.1.1 RC.

As the 4.x release manager, I really don't want to get into the
business of selecting what should and should not go into a particular
release. Every 4-6 weeks, we should just cut a new minor release
branch off master and take everything that was contributed during that
time, as long as everything builds fine and all the tests pass. The
master branch, while it doesn't always have to be perfect, should at
least always be deployable. If a new feature is merged with bugs and
becomes a show-stopper, it should be either immediately remedied or
reverted on master while we work on a fix.

- Rawlin

On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
<Mi...@comcast.com> wrote:
>
> In my opinion, in order to get to a cadence we are discussing we need to put a lot more work into the CI system. It has been failing consistently, doesn’t block PR approvals/merges and when it actually runs it does not test anything besides build and license headers.
>
> In the past couple of weeks we have had (off the top of my head):
>
> * Unit tests broken
> * TO API tests broken
> * Golang vendor issues
> * TO Go build issues
>
> All were merged and broke master.
>
> Yes, having release branches sounds great and more aggressive cadence will minimize the amount of risk but I think that comes with the need to improve on our automated validation and testing.
>
> We have proven that we have not kept master in a good state and adding more releases (more branches) will make it even harder to keep that stability we are already fighting.
>
> On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
>
>     Yeah, I get it. No one company should be driving release schedules/scope.
>
>     What I was really getting at is if ANY company has a recent version in a
>     test or prod-like enviro (meaning the version is being exercised and
>     thoroughly tested), then maybe we consider simply cutting a release from
>     that version. For example, imagine company X had this version in a
>     test/prod enviro:
>
>     Master-10287.7e62d07 (
>     https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master)
>
>     Then I would be in favor of cutting the 4.0 release from
>     Master-10287.7e62d07 as we know it is proven to work based on company X's
>     actual use of it.
>
>     The truth is (imo), we don't have the bandwidth to manually verify releases
>     that we are not using/have not used nor do we have the necessary automated
>     test coverage to verify these releases. So most of our releases are
>     significantly untested.
>
>     On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com> wrote:
>
>     > I'm really not a fan of allowing Comcast to dictate the release scope and
>     > schedule. If cherry-picking is too messy, then we can just cut new minor
>     > releases directly from master.
>     >
>     > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com> wrote:
>     >
>     > > I don't think it's as easy as cherry picking (backporting) certain
>     > features
>     > > into a release branch. I could be wrong but I really don't think it is.
>     > >
>     > > So what I'm hearing is that 4.0 gets cut from master and we go through
>     > with
>     > > our normal process of testing, validating, etc. At the same time, 4.1
>     > > branch is created from 4.0. Master moves on as normal. Let's say 25
>     > commits
>     > > come in to master during the next 6 weeks. During that 6 weeks, we cherry
>     > > pick only the features that look interesting for 4.1? (the rest just stay
>     > > in master and will get picked up in 5.0?)
>     > >
>     > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart
>     > and
>     > > have only "interesting" features so the releases are incrementally very
>     > > small and much easier to test/validate, however, those features might
>     > > depend on other commits. At some point, cherry picking becomes impossible
>     > > as the foundation of the code base has shifted so drastically. Cherry
>     > > picking will become very messy and introduce a high level of risk imo.
>     > >
>     > > Also, you'd have to manage the scope of these minor releases which we are
>     > > not very good at because the designated release manager has a full-time
>     > job
>     > > to attend to and now we have even more releases to
>     > > scope/test/validate/approve. I like the idea of faster releases but this
>     > > sounds like more work that nobody has the bandwidth for.
>     > >
>     > > Ok, so rather than shooting down an idea with no alternative proposal, I
>     > > propose this. At Comcast, we are trying to get to the point where we
>     > > release TC to prod from the head of master every few weeks. What if every
>     > > time Comcast does an internal upgrade, they note the commit hash and once
>     > > comcast is happy with the upgrade in their prod enviro, they propose a
>     > > release pinned to that commit hash? With this approach you'd end up with:
>     > >
>     > > 1. fast releases (hopefully every few weeks)
>     > > 2. well tested releases as it's in a real prod enviro
>     > > 3. little/no validation work for anyone
>     > >
>     > > Also, this doesn't have to be Comcast. If anyone else is out front at the
>     > > edge of master and running it in a trusted enviro, they could propose a
>     > > release as well.
>     > >
>     > > Jeremy
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <da...@gmail.com>
>     > > wrote:
>     > >
>     > > > I have been thinking about how we can get better with our release
>     > > cadence.
>     > > > I feel like we have slowed to a crawl and not been as good as we should
>     > > > about how we release.  Our last Major release was in March and we
>     > haven't
>     > > > had a real release since.   Moving forward I would like to see us get
>     > on
>     > > a
>     > > > more consistent cadence and provide smaller releases that can be more
>     > > > easily digested by the community.  It's my hope that we can get to a
>     > > > cadence where we are doing releases every 4 to 6 weeks,  we are
>     > releasing
>     > > > in such a way that not all releases all required (e.g. if you are on
>     > 4.0
>     > > > you don't need 4.1, you can just install 4.2), and we are more
>     > deliberate
>     > > > about what we put into a release.
>     > > >
>     > > > I think we can accomplish this by using our release branches better.
>     > For
>     > > > each major release we will cut a release branch from master (e.g. 4.x)
>     > > and
>     > > > then we will use cherry-picking to add new features and bug fixes to
>     > that
>     > > > release.  This means that if 4.0 has been released and you want to get
>     > > your
>     > > > feature/bug fix in 4.1, you will first submit your PR to master and
>     > then
>     > > > cheery pick your squash merged commit to the 4.x branch which we will
>     > use
>     > > > to create the 4.1 release.  I think we should allow either the
>     > > contributor
>     > > > or the committer (who is merging the PR) to suggest if a feature goes
>     > > into
>     > > > the release branch, and if we disagree we can take it to the list.  If
>     > we
>     > > > decide that every PR to master also goes into 4.1 then so be it, but at
>     > > > least we are making a conscious decision of what goes into the next
>     > > > release.   This will allow us to not be so worried about what we are
>     > > > merging into master and how that will affect our next release.
>     > According
>     > > > to our 6 week cadence we will cut a new release from the current
>     > feature
>     > > > branch and put that up for a vote.  These releases will be small and
>     > > > testable enough that we can get feedback quickly and move from RC to
>     > > > release in a short time.  If a release has too many issues then we may
>     > > end
>     > > > up deciding to skip that minor release in favor of the next one,
>     > > hopefully
>     > > > this does not happen very often if at all.
>     > > >
>     > > > Once a breaking change is introduced to master which will require a new
>     > > > major release, we will create a new major release branch from master
>     > > (which
>     > > > will mean a new release manager).  We will then repeat the same process
>     > > > with the new release branch, etc, etc.
>     > > >
>     > > > As for LTS we will provide support for the latest major release plus
>     > the
>     > > > one previous.  So, once we release 4.0 we will support 4.0 and 3.1.
>     > Any
>     > > > security issues that arise will -- if present -- be applied to each of
>     > > > these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc,
>     > > > etc.
>     > > >
>     > > > Please let me know if you have any thoughts on this plan.  If there are
>     > > no
>     > > > major objections I would like to try this with 4.x with the idea that
>     > we
>     > > > will adjust how we do things as necessary.
>     > > >
>     > > > We need to get better at releasing and something needs to change,
>     > > hopefully
>     > > > this can get us going in the right direction.
>     > > >
>     > > > Thanks,
>     > > > Dave
>     > > >
>     > > >
>     > > > TL;DR -  I am proposing a 6 week release cycle using cherry-picks to
>     > our
>     > > > release branch to control what goes into a minor release.  Major
>     > releases
>     > > > will be cut from master as necessary - such as if there is a breaking
>     > > > change that is introduced.
>     > > >
>     > >
>     >
>
>

Re: [EXTERNAL] Re: Release Process

["Hoppal, Michael" <Mi...@comcast.com>]

In my opinion, in order to get to a cadence we are discussing we need to put a lot more work into the CI system. It has been failing consistently, doesn’t block PR approvals/merges and when it actually runs it does not test anything besides build and license headers.

In the past couple of weeks we have had (off the top of my head):

* Unit tests broken
* TO API tests broken
* Golang vendor issues
* TO Go build issues

All were merged and broke master.

Yes, having release branches sounds great and more aggressive cadence will minimize the amount of risk but I think that comes with the need to improve on our automated validation and testing.

We have proven that we have not kept master in a good state and adding more releases (more branches) will make it even harder to keep that stability we are already fighting.

On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:

    Yeah, I get it. No one company should be driving release schedules/scope.

    What I was really getting at is if ANY company has a recent version in a
    test or prod-like enviro (meaning the version is being exercised and
    thoroughly tested), then maybe we consider simply cutting a release from
    that version. For example, imagine company X had this version in a
    test/prod enviro:

    Master-10287.7e62d07 (
    https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master)

    Then I would be in favor of cutting the 4.0 release from
    Master-10287.7e62d07 as we know it is proven to work based on company X's
    actual use of it.

    The truth is (imo), we don't have the bandwidth to manually verify releases
    that we are not using/have not used nor do we have the necessary automated
    test coverage to verify these releases. So most of our releases are
    significantly untested.

    On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com> wrote:

    > I'm really not a fan of allowing Comcast to dictate the release scope and
    > schedule. If cherry-picking is too messy, then we can just cut new minor
    > releases directly from master.
    >
    > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com> wrote:
    >
    > > I don't think it's as easy as cherry picking (backporting) certain
    > features
    > > into a release branch. I could be wrong but I really don't think it is.
    > >
    > > So what I'm hearing is that 4.0 gets cut from master and we go through
    > with
    > > our normal process of testing, validating, etc. At the same time, 4.1
    > > branch is created from 4.0. Master moves on as normal. Let's say 25
    > commits
    > > come in to master during the next 6 weeks. During that 6 weeks, we cherry
    > > pick only the features that look interesting for 4.1? (the rest just stay
    > > in master and will get picked up in 5.0?)
    > >
    > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart
    > and
    > > have only "interesting" features so the releases are incrementally very
    > > small and much easier to test/validate, however, those features might
    > > depend on other commits. At some point, cherry picking becomes impossible
    > > as the foundation of the code base has shifted so drastically. Cherry
    > > picking will become very messy and introduce a high level of risk imo.
    > >
    > > Also, you'd have to manage the scope of these minor releases which we are
    > > not very good at because the designated release manager has a full-time
    > job
    > > to attend to and now we have even more releases to
    > > scope/test/validate/approve. I like the idea of faster releases but this
    > > sounds like more work that nobody has the bandwidth for.
    > >
    > > Ok, so rather than shooting down an idea with no alternative proposal, I
    > > propose this. At Comcast, we are trying to get to the point where we
    > > release TC to prod from the head of master every few weeks. What if every
    > > time Comcast does an internal upgrade, they note the commit hash and once
    > > comcast is happy with the upgrade in their prod enviro, they propose a
    > > release pinned to that commit hash? With this approach you'd end up with:
    > >
    > > 1. fast releases (hopefully every few weeks)
    > > 2. well tested releases as it's in a real prod enviro
    > > 3. little/no validation work for anyone
    > >
    > > Also, this doesn't have to be Comcast. If anyone else is out front at the
    > > edge of master and running it in a trusted enviro, they could propose a
    > > release as well.
    > >
    > > Jeremy
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <da...@gmail.com>
    > > wrote:
    > >
    > > > I have been thinking about how we can get better with our release
    > > cadence.
    > > > I feel like we have slowed to a crawl and not been as good as we should
    > > > about how we release.  Our last Major release was in March and we
    > haven't
    > > > had a real release since.   Moving forward I would like to see us get
    > on
    > > a
    > > > more consistent cadence and provide smaller releases that can be more
    > > > easily digested by the community.  It's my hope that we can get to a
    > > > cadence where we are doing releases every 4 to 6 weeks,  we are
    > releasing
    > > > in such a way that not all releases all required (e.g. if you are on
    > 4.0
    > > > you don't need 4.1, you can just install 4.2), and we are more
    > deliberate
    > > > about what we put into a release.
    > > >
    > > > I think we can accomplish this by using our release branches better.
    > For
    > > > each major release we will cut a release branch from master (e.g. 4.x)
    > > and
    > > > then we will use cherry-picking to add new features and bug fixes to
    > that
    > > > release.  This means that if 4.0 has been released and you want to get
    > > your
    > > > feature/bug fix in 4.1, you will first submit your PR to master and
    > then
    > > > cheery pick your squash merged commit to the 4.x branch which we will
    > use
    > > > to create the 4.1 release.  I think we should allow either the
    > > contributor
    > > > or the committer (who is merging the PR) to suggest if a feature goes
    > > into
    > > > the release branch, and if we disagree we can take it to the list.  If
    > we
    > > > decide that every PR to master also goes into 4.1 then so be it, but at
    > > > least we are making a conscious decision of what goes into the next
    > > > release.   This will allow us to not be so worried about what we are
    > > > merging into master and how that will affect our next release.
    > According
    > > > to our 6 week cadence we will cut a new release from the current
    > feature
    > > > branch and put that up for a vote.  These releases will be small and
    > > > testable enough that we can get feedback quickly and move from RC to
    > > > release in a short time.  If a release has too many issues then we may
    > > end
    > > > up deciding to skip that minor release in favor of the next one,
    > > hopefully
    > > > this does not happen very often if at all.
    > > >
    > > > Once a breaking change is introduced to master which will require a new
    > > > major release, we will create a new major release branch from master
    > > (which
    > > > will mean a new release manager).  We will then repeat the same process
    > > > with the new release branch, etc, etc.
    > > >
    > > > As for LTS we will provide support for the latest major release plus
    > the
    > > > one previous.  So, once we release 4.0 we will support 4.0 and 3.1.
    > Any
    > > > security issues that arise will -- if present -- be applied to each of
    > > > these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc,
    > > > etc.
    > > >
    > > > Please let me know if you have any thoughts on this plan.  If there are
    > > no
    > > > major objections I would like to try this with 4.x with the idea that
    > we
    > > > will adjust how we do things as necessary.
    > > >
    > > > We need to get better at releasing and something needs to change,
    > > hopefully
    > > > this can get us going in the right direction.
    > > >
    > > > Thanks,
    > > > Dave
    > > >
    > > >
    > > > TL;DR -  I am proposing a 6 week release cycle using cherry-picks to
    > our
    > > > release branch to control what goes into a minor release.  Major
    > releases
    > > > will be cut from master as necessary - such as if there is a breaking
    > > > change that is introduced.
    > > >
    > >
    >

Re: Release Process

[Jeremy Mitchell <mi...@gmail.com>]

Yeah, I get it. No one company should be driving release schedules/scope.

What I was really getting at is if ANY company has a recent version in a
test or prod-like enviro (meaning the version is being exercised and
thoroughly tested), then maybe we consider simply cutting a release from
that version. For example, imagine company X had this version in a
test/prod enviro:

Master-10287.7e62d07 (
https://github.com/apache/trafficcontrol/commits/master)

Then I would be in favor of cutting the 4.0 release from
Master-10287.7e62d07 as we know it is proven to work based on company X's
actual use of it.

The truth is (imo), we don't have the bandwidth to manually verify releases
that we are not using/have not used nor do we have the necessary automated
test coverage to verify these releases. So most of our releases are
significantly untested.

On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <oc...@gmail.com> wrote:

> I'm really not a fan of allowing Comcast to dictate the release scope and
> schedule. If cherry-picking is too messy, then we can just cut new minor
> releases directly from master.
>
> On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com> wrote:
>
> > I don't think it's as easy as cherry picking (backporting) certain
> features
> > into a release branch. I could be wrong but I really don't think it is.
> >
> > So what I'm hearing is that 4.0 gets cut from master and we go through
> with
> > our normal process of testing, validating, etc. At the same time, 4.1
> > branch is created from 4.0. Master moves on as normal. Let's say 25
> commits
> > come in to master during the next 6 weeks. During that 6 weeks, we cherry
> > pick only the features that look interesting for 4.1? (the rest just stay
> > in master and will get picked up in 5.0?)
> >
> > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart
> and
> > have only "interesting" features so the releases are incrementally very
> > small and much easier to test/validate, however, those features might
> > depend on other commits. At some point, cherry picking becomes impossible
> > as the foundation of the code base has shifted so drastically. Cherry
> > picking will become very messy and introduce a high level of risk imo.
> >
> > Also, you'd have to manage the scope of these minor releases which we are
> > not very good at because the designated release manager has a full-time
> job
> > to attend to and now we have even more releases to
> > scope/test/validate/approve. I like the idea of faster releases but this
> > sounds like more work that nobody has the bandwidth for.
> >
> > Ok, so rather than shooting down an idea with no alternative proposal, I
> > propose this. At Comcast, we are trying to get to the point where we
> > release TC to prod from the head of master every few weeks. What if every
> > time Comcast does an internal upgrade, they note the commit hash and once
> > comcast is happy with the upgrade in their prod enviro, they propose a
> > release pinned to that commit hash? With this approach you'd end up with:
> >
> > 1. fast releases (hopefully every few weeks)
> > 2. well tested releases as it's in a real prod enviro
> > 3. little/no validation work for anyone
> >
> > Also, this doesn't have to be Comcast. If anyone else is out front at the
> > edge of master and running it in a trusted enviro, they could propose a
> > release as well.
> >
> > Jeremy
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <da...@gmail.com>
> > wrote:
> >
> > > I have been thinking about how we can get better with our release
> > cadence.
> > > I feel like we have slowed to a crawl and not been as good as we should
> > > about how we release.  Our last Major release was in March and we
> haven't
> > > had a real release since.   Moving forward I would like to see us get
> on
> > a
> > > more consistent cadence and provide smaller releases that can be more
> > > easily digested by the community.  It's my hope that we can get to a
> > > cadence where we are doing releases every 4 to 6 weeks,  we are
> releasing
> > > in such a way that not all releases all required (e.g. if you are on
> 4.0
> > > you don't need 4.1, you can just install 4.2), and we are more
> deliberate
> > > about what we put into a release.
> > >
> > > I think we can accomplish this by using our release branches better.
> For
> > > each major release we will cut a release branch from master (e.g. 4.x)
> > and
> > > then we will use cherry-picking to add new features and bug fixes to
> that
> > > release.  This means that if 4.0 has been released and you want to get
> > your
> > > feature/bug fix in 4.1, you will first submit your PR to master and
> then
> > > cheery pick your squash merged commit to the 4.x branch which we will
> use
> > > to create the 4.1 release.  I think we should allow either the
> > contributor
> > > or the committer (who is merging the PR) to suggest if a feature goes
> > into
> > > the release branch, and if we disagree we can take it to the list.  If
> we
> > > decide that every PR to master also goes into 4.1 then so be it, but at
> > > least we are making a conscious decision of what goes into the next
> > > release.   This will allow us to not be so worried about what we are
> > > merging into master and how that will affect our next release.
> According
> > > to our 6 week cadence we will cut a new release from the current
> feature
> > > branch and put that up for a vote.  These releases will be small and
> > > testable enough that we can get feedback quickly and move from RC to
> > > release in a short time.  If a release has too many issues then we may
> > end
> > > up deciding to skip that minor release in favor of the next one,
> > hopefully
> > > this does not happen very often if at all.
> > >
> > > Once a breaking change is introduced to master which will require a new
> > > major release, we will create a new major release branch from master
> > (which
> > > will mean a new release manager).  We will then repeat the same process
> > > with the new release branch, etc, etc.
> > >
> > > As for LTS we will provide support for the latest major release plus
> the
> > > one previous.  So, once we release 4.0 we will support 4.0 and 3.1.
> Any
> > > security issues that arise will -- if present -- be applied to each of
> > > these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc,
> > > etc.
> > >
> > > Please let me know if you have any thoughts on this plan.  If there are
> > no
> > > major objections I would like to try this with 4.x with the idea that
> we
> > > will adjust how we do things as necessary.
> > >
> > > We need to get better at releasing and something needs to change,
> > hopefully
> > > this can get us going in the right direction.
> > >
> > > Thanks,
> > > Dave
> > >
> > >
> > > TL;DR -  I am proposing a 6 week release cycle using cherry-picks to
> our
> > > release branch to control what goes into a minor release.  Major
> releases
> > > will be cut from master as necessary - such as if there is a breaking
> > > change that is introduced.
> > >
> >
>

Re: Release Process

[ocket 8888 <oc...@gmail.com>]

I'm really not a fan of allowing Comcast to dictate the release scope and
schedule. If cherry-picking is too messy, then we can just cut new minor
releases directly from master.

On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mi...@gmail.com> wrote:

> I don't think it's as easy as cherry picking (backporting) certain features
> into a release branch. I could be wrong but I really don't think it is.
>
> So what I'm hearing is that 4.0 gets cut from master and we go through with
> our normal process of testing, validating, etc. At the same time, 4.1
> branch is created from 4.0. Master moves on as normal. Let's say 25 commits
> come in to master during the next 6 weeks. During that 6 weeks, we cherry
> pick only the features that look interesting for 4.1? (the rest just stay
> in master and will get picked up in 5.0?)
>
> In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart and
> have only "interesting" features so the releases are incrementally very
> small and much easier to test/validate, however, those features might
> depend on other commits. At some point, cherry picking becomes impossible
> as the foundation of the code base has shifted so drastically. Cherry
> picking will become very messy and introduce a high level of risk imo.
>
> Also, you'd have to manage the scope of these minor releases which we are
> not very good at because the designated release manager has a full-time job
> to attend to and now we have even more releases to
> scope/test/validate/approve. I like the idea of faster releases but this
> sounds like more work that nobody has the bandwidth for.
>
> Ok, so rather than shooting down an idea with no alternative proposal, I
> propose this. At Comcast, we are trying to get to the point where we
> release TC to prod from the head of master every few weeks. What if every
> time Comcast does an internal upgrade, they note the commit hash and once
> comcast is happy with the upgrade in their prod enviro, they propose a
> release pinned to that commit hash? With this approach you'd end up with:
>
> 1. fast releases (hopefully every few weeks)
> 2. well tested releases as it's in a real prod enviro
> 3. little/no validation work for anyone
>
> Also, this doesn't have to be Comcast. If anyone else is out front at the
> edge of master and running it in a trusted enviro, they could propose a
> release as well.
>
> Jeremy
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Oct 29, 2019 at 3:06 PM David Neuman <da...@gmail.com>
> wrote:
>
> > I have been thinking about how we can get better with our release
> cadence.
> > I feel like we have slowed to a crawl and not been as good as we should
> > about how we release.  Our last Major release was in March and we haven't
> > had a real release since.   Moving forward I would like to see us get on
> a
> > more consistent cadence and provide smaller releases that can be more
> > easily digested by the community.  It's my hope that we can get to a
> > cadence where we are doing releases every 4 to 6 weeks,  we are releasing
> > in such a way that not all releases all required (e.g. if you are on 4.0
> > you don't need 4.1, you can just install 4.2), and we are more deliberate
> > about what we put into a release.
> >
> > I think we can accomplish this by using our release branches better.  For
> > each major release we will cut a release branch from master (e.g. 4.x)
> and
> > then we will use cherry-picking to add new features and bug fixes to that
> > release.  This means that if 4.0 has been released and you want to get
> your
> > feature/bug fix in 4.1, you will first submit your PR to master and then
> > cheery pick your squash merged commit to the 4.x branch which we will use
> > to create the 4.1 release.  I think we should allow either the
> contributor
> > or the committer (who is merging the PR) to suggest if a feature goes
> into
> > the release branch, and if we disagree we can take it to the list.  If we
> > decide that every PR to master also goes into 4.1 then so be it, but at
> > least we are making a conscious decision of what goes into the next
> > release.   This will allow us to not be so worried about what we are
> > merging into master and how that will affect our next release.  According
> > to our 6 week cadence we will cut a new release from the current feature
> > branch and put that up for a vote.  These releases will be small and
> > testable enough that we can get feedback quickly and move from RC to
> > release in a short time.  If a release has too many issues then we may
> end
> > up deciding to skip that minor release in favor of the next one,
> hopefully
> > this does not happen very often if at all.
> >
> > Once a breaking change is introduced to master which will require a new
> > major release, we will create a new major release branch from master
> (which
> > will mean a new release manager).  We will then repeat the same process
> > with the new release branch, etc, etc.
> >
> > As for LTS we will provide support for the latest major release plus the
> > one previous.  So, once we release 4.0 we will support 4.0 and 3.1.  Any
> > security issues that arise will -- if present -- be applied to each of
> > these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc,
> > etc.
> >
> > Please let me know if you have any thoughts on this plan.  If there are
> no
> > major objections I would like to try this with 4.x with the idea that we
> > will adjust how we do things as necessary.
> >
> > We need to get better at releasing and something needs to change,
> hopefully
> > this can get us going in the right direction.
> >
> > Thanks,
> > Dave
> >
> >
> > TL;DR -  I am proposing a 6 week release cycle using cherry-picks to our
> > release branch to control what goes into a minor release.  Major releases
> > will be cut from master as necessary - such as if there is a breaking
> > change that is introduced.
> >
>

Re: Release Process

[Jeremy Mitchell <mi...@gmail.com>]

I don't think it's as easy as cherry picking (backporting) certain features
into a release branch. I could be wrong but I really don't think it is.

So what I'm hearing is that 4.0 gets cut from master and we go through with
our normal process of testing, validating, etc. At the same time, 4.1
branch is created from 4.0. Master moves on as normal. Let's say 25 commits
come in to master during the next 6 weeks. During that 6 weeks, we cherry
pick only the features that look interesting for 4.1? (the rest just stay
in master and will get picked up in 5.0?)

In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart and
have only "interesting" features so the releases are incrementally very
small and much easier to test/validate, however, those features might
depend on other commits. At some point, cherry picking becomes impossible
as the foundation of the code base has shifted so drastically. Cherry
picking will become very messy and introduce a high level of risk imo.

Also, you'd have to manage the scope of these minor releases which we are
not very good at because the designated release manager has a full-time job
to attend to and now we have even more releases to
scope/test/validate/approve. I like the idea of faster releases but this
sounds like more work that nobody has the bandwidth for.

Ok, so rather than shooting down an idea with no alternative proposal, I
propose this. At Comcast, we are trying to get to the point where we
release TC to prod from the head of master every few weeks. What if every
time Comcast does an internal upgrade, they note the commit hash and once
comcast is happy with the upgrade in their prod enviro, they propose a
release pinned to that commit hash? With this approach you'd end up with:

1. fast releases (hopefully every few weeks)
2. well tested releases as it's in a real prod enviro
3. little/no validation work for anyone

Also, this doesn't have to be Comcast. If anyone else is out front at the
edge of master and running it in a trusted enviro, they could propose a
release as well.

Jeremy
















On Tue, Oct 29, 2019 at 3:06 PM David Neuman <da...@gmail.com>
wrote:

> I have been thinking about how we can get better with our release cadence.
> I feel like we have slowed to a crawl and not been as good as we should
> about how we release.  Our last Major release was in March and we haven't
> had a real release since.   Moving forward I would like to see us get on a
> more consistent cadence and provide smaller releases that can be more
> easily digested by the community.  It's my hope that we can get to a
> cadence where we are doing releases every 4 to 6 weeks,  we are releasing
> in such a way that not all releases all required (e.g. if you are on 4.0
> you don't need 4.1, you can just install 4.2), and we are more deliberate
> about what we put into a release.
>
> I think we can accomplish this by using our release branches better.  For
> each major release we will cut a release branch from master (e.g. 4.x) and
> then we will use cherry-picking to add new features and bug fixes to that
> release.  This means that if 4.0 has been released and you want to get your
> feature/bug fix in 4.1, you will first submit your PR to master and then
> cheery pick your squash merged commit to the 4.x branch which we will use
> to create the 4.1 release.  I think we should allow either the contributor
> or the committer (who is merging the PR) to suggest if a feature goes into
> the release branch, and if we disagree we can take it to the list.  If we
> decide that every PR to master also goes into 4.1 then so be it, but at
> least we are making a conscious decision of what goes into the next
> release.   This will allow us to not be so worried about what we are
> merging into master and how that will affect our next release.  According
> to our 6 week cadence we will cut a new release from the current feature
> branch and put that up for a vote.  These releases will be small and
> testable enough that we can get feedback quickly and move from RC to
> release in a short time.  If a release has too many issues then we may end
> up deciding to skip that minor release in favor of the next one, hopefully
> this does not happen very often if at all.
>
> Once a breaking change is introduced to master which will require a new
> major release, we will create a new major release branch from master (which
> will mean a new release manager).  We will then repeat the same process
> with the new release branch, etc, etc.
>
> As for LTS we will provide support for the latest major release plus the
> one previous.  So, once we release 4.0 we will support 4.0 and 3.1.  Any
> security issues that arise will -- if present -- be applied to each of
> these versions.  Once 4.1 is released we will support 4.1 and 3.1, etc,
> etc.
>
> Please let me know if you have any thoughts on this plan.  If there are no
> major objections I would like to try this with 4.x with the idea that we
> will adjust how we do things as necessary.
>
> We need to get better at releasing and something needs to change, hopefully
> this can get us going in the right direction.
>
> Thanks,
> Dave
>
>
> TL;DR -  I am proposing a 6 week release cycle using cherry-picks to our
> release branch to control what goes into a minor release.  Major releases
> will be cut from master as necessary - such as if there is a breaking
> change that is introduced.
>