You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Schmidt, Andrew (Contractor)" <An...@comcast.com> on 2019/11/01 17:39:31 UTC
Re: [EXTERNAL] Re: Release Process
I'm +1 on this final summary. It infers that we are going to soon be adding a job that runs CI checks on Master. This is good. That's probably a subject for another thread. I do have one question about this approach. I think there is probably a scenario where we need to fix an issue on the 4.x branch itself. In this case I guess we would merge that commit back to master? And then RM would have to keep track of when its safe to fast-forward to the merged version of the fix?
On 10/31/19, 4:26 PM, "Dave Neuman" <ne...@apache.org> wrote:
Yeah, I think that about sums it up. Hopefully we can help the release
manager with some of those responsibilities.
On Thu, Oct 31, 2019 at 2:31 PM Rawlin Peters <ra...@apache.org> wrote:
> Alright, I think that sounds fair. In general what I'm hearing is,
> every 4-6 weeks:
> 1. fast-forward 4.x to the most recent "stable" commit of master
> (maybe we automate this and fast-forward every day if some set of
> tests pass, unless we explicitly know master is unstable for other
> reasons)
> 2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
> 4.*.x that are necessary for the release (e.g. bugs that break the
> data plane or prevent operators from being able to do their job or run
> the control plane)
>
> Which means the release manager needs to know/track:
> - unstable commits along with their corresponding fixes
> - incomplete features
> - features we'd like to hold until the next release (due to current
> release having enough already)
> - any cherry-picks that were deemed necessary for a certain release
> (so that they can either be carried forward into the next minor
> release or, ideally, included as part of fast-forwarding 4.x to the
> most recent stable commit on master)
>
> I think this will work. We gain the ability to:
> 1. let master be "unstable" periodically
> 2. control the size of releases
> 3. make releases mirror master as closely as possible (because they
> will just be a snapshot of the most recent version of master that was
> considered stable -- unless absolutely necessary bug/security fixes
> were cherry-picked in)
>
> Does that all sound right, Dave?
>
> - Rawlin
>
>
>
>
>
>
>
> On Thu, Oct 31, 2019 at 8:37 AM David Neuman <da...@gmail.com>
> wrote:
> >
> > I think a third reason would be that we are getting very close to our
> > release branch date, it's a large/complex feature, and we feel like the
> > release already has enough in it. In that case we could hold off on
> adding
> > that change to the 4.x branch until after we cut the 4.x.x branch. The
> > benefit is that we can still merge that feature into master and get all
> of
> > our testing (both internally and externally) running against it. If you
> > stick with just branching off master and this situation arises then we
> get
> > in the spot where we just don't merge the PR at all until after the
> release
> > branch is cut (we have two examples of this right now). I personally
> think
> > it's better to get this into master and let our automation run against it
> > than it is to leave it out in a PR even though it is ready.
> >
> > As for the process, I was thinking the same thing except that you don't
> > even need to cherry-pick. In your scenario above you can just rebase
> > master into 4.x once E was merged. This avoids the headaches of having
> to
> > cherry pick each commit while still giving us the benefit of having all
> of
> > the changes on the release branch. We can even decide to setup
> automation
> > to rebase master into the release branch on an interval or on demand. If
> > we rebase in something that turns out to be broken we just roll back to
> the
> > previous rebase (or a certain commit) until that is fixed and then rebase
> > again. If a feature gets merged to master that we aren't ready to put
> into
> > the release, then we stop the automated rebase until we are ready to get
> it
> > into the release -- at that point we should consider ONLY cherry-picking
> > bug fixes for the remainder of that release cycle to make things easier
> to
> > rebase again.
> >
> > I am sure there will be unintended consequences, and I am sure we will
> have
> > challenges to work through, but I still think that it's worth trying. We
> > should fail fast and be willing to change course if necessary.
> > I think if we can be consistent enough with our releases then maybe we
> can
> > discuss branching from master again.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > I guess my concern is really about what possible reasons we might have
> > > to not cherry-pick a specific PR that had been merged into master
> > > already. The only reasons I can think of to not cherry-pick something
> > > would be:
> > > 1. the PR was completely broken (e.g. breaks the build, tests don't
> > > pass, doesn't compile, etc).
> > > 2. the PR one one of a larger multi-PR "feature", and *we know*
> > > releasing some parts early could cause problems (that knowing part is
> > > crucial)
> > >
> > > We should always strive to catch issue #1 in our CI before it merges
> > > into master. When things like that merge it can block people from
> > > doing development starting at that point until it's fixed.
> > > I can see why issue #2 might arise due to multiple different people
> > > developing one larger feature, but we should still strive to break
> > > larger features up into individual pieces that can reasonably stand on
> > > their own if possible. Due to time constraints or other reasons I can
> > > see that might be a little difficult.
> > >
> > > That said, I would still strive to make the 4.x release branch a
> > > snapshot of master at some "known good point" in time. Let's say we
> > > cut 4.x at commit A, then a broken commit B gets merged into master.
> > > If it's completely obvious that it's a broken commit, I'd wait until
> > > at least a commit C gets merged to fix it. Then I would cherry-pick B
> > > and C into 4.x at the same time. If the fix is really like a commit E
> > > with unrelated commits C and D in between, ideally I would still wait
> > > to cherry-pick C and D until a "fix" commit E was merged (to fix
> > > commit B), at which point I'd cherrypick commits B through E at the
> > > same time, in order, to avoid any possible merge conflicts.
> > >
> > > I think to make that process easier for myself I would just
> > > cherry-pick everything into the 4.x branch as it gets merged into
> > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
> > > commits were not fixed/completed in time, I would reset the 4.x branch
> > > to the last stable commit. This would mean that unstable
> > > commits/incomplete features could potentially block things from going
> > > into the release. I might count that as a positive because it would
> > > incentivize us to improve our CI and do our best to also keep the
> > > master branch as stable as possible.
> > >
> > > The end result would be the same as just continually jumping up the
> > > master branch from known good, stable points in time to the next,
> > > creating a minor release branch at each of those points.
> > >
> > > - Rawlin
> > >
> > >
> > >
> > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <ne...@apache.org> wrote:
> > > >
> > > > Sorry, I have taken too long to respond.
> > > >
> > > > +1000000 on better CI and Testing, I think we all agree that needs
> to get
> > > > better and is a different beast than this topic.
> > > >
> > > > As for the release branches, I am trying to follow a process similar
> to
> > > > ATS, which is usually the model we try to follow with our project.
> > > > While I agree that branching from master is the ideal solution, we
> tried
> > > > that and it didn't work. I feel like we are trying to agree to do
> the
> > > same
> > > > thing that we already tried and didn't work. I think we should have
> a
> > > > little freedom to commit things into master and, if they break
> things,
> > > work
> > > > on getting them fixed. We also might commit PRs to master that are
> > > partial
> > > > implementations of a bigger feature. If we are using a time based
> > > > strategy to cut our releases from master then we are going to be
> > > > continually worrying about making sure Master stays not broken. I
> think
> > > we
> > > > should try to keep master deployable, but it's fine if it's not for
> short
> > > > periods of time.
> > > >
> > > > I am not trying to suggest that we are "picky" about what goes into
> our
> > > > release branch. I think we should try to cherry-pick as much as we
> can
> > > > into our release branch once it is ready to be cherry-picked. I
> think we
> > > > should be able to take the tip of our release branch and deploy it
> > > whenever
> > > > we want with some confidence since we know that features/bug-fixes
> that
> > > > have been cherry-picked have already been reviewed and tested against
> > > > master. Yes, its more work, but it also give us the ability to have
> more
> > > > confidence in what is in our release at any given time. Again, we
> should
> > > > be trying to cherry-pick in everything that makes sense to
> cherry-pick.
> > > >
> > > > I think the process that I would like to see looks like this:
> > > > - Cut a 4.x branch from Master
> > > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > > branch
> > > > - Tag the 4.0.0 release from the 4.0.x branch
> > > > - New non-breaking changes and bug fixes that go into master are also
> > > > Cherry Picked to 4.x
> > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag
> when we
> > > > are ready for a RC.
> > > > - Rinse and repeat for all 4.x releases
> > > > - Once there's a need for a major release we make a 5.x branch from
> > > master
> > > >
> > > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > > then
> > > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > > have
> > > > 4.1 yet). Yes, that is a more work than today, but I think A) it's
> not
> > > > really that bad and B) it gives us more control.
> > > >
> > > > This should allow us to deploy much quicker and with more confidence
> than
> > > > is possible today. We can also choose to deploy from any point in
> time
> > > > from the 4.x branch with relatively high confidence.
> > > >
> > > > I'll wrap this up by saying that I don't think there's a perfect
> > > solution,
> > > > but I think we can agree on something that is worth trying at least
> with
> > > > the 4.x release life cycle. Once we have some experience, we can
> figure
> > > > out where it is not meeting our needs and figure out how to adjust as
> > > > necessary. I think we can all agree our current process is not
> working.
> > > >
> > > > Thanks,
> > > > Dave
> > > >
> > > >
> > > >
> > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <ra...@apache.org>
> > > wrote:
> > > >
> > > > > +1 on more frequent releases
> > > > > +1 on improving our CI. We need to get our CI to the point where
> we're
> > > > > confident deploying changes to production if the CI passes.
> > > > > +1 on "trunk-based" development as opposed to GitFlow
> > > > > -1 on cherry-picking new features. This is risky, tedious, and
> > > > > represents unnecessary effort that would be better spent elsewhere,
> > > > > like improving our CI or adding more automated tests.
> > > > > +1 on cutting minor release branches directly off master
> > > > >
> > > > > I think cutting minor release branches directly off master can put
> us
> > > > > in a win-win scenario. Our releases become smaller and more
> frequent,
> > > > > and we avoid the unnecessary risk and effort of cherry-picking new
> > > > > features onto stale release branches.
> > > > >
> > > > > This is the process that I would propose:
> > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > > > > 2. Develop master for 4-6 weeks.
> > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > > > > 5. Whenever a major (show-stopping) bug or security issue is found,
> > > > > cherry-pick the fix from master into the latest release branch
> (e.g.
> > > > > 4.1.x), and create a 4.1.1 RC.
> > > > >
> > > > > As the 4.x release manager, I really don't want to get into the
> > > > > business of selecting what should and should not go into a
> particular
> > > > > release. Every 4-6 weeks, we should just cut a new minor release
> > > > > branch off master and take everything that was contributed during
> that
> > > > > time, as long as everything builds fine and all the tests pass. The
> > > > > master branch, while it doesn't always have to be perfect, should
> at
> > > > > least always be deployable. If a new feature is merged with bugs
> and
> > > > > becomes a show-stopper, it should be either immediately remedied or
> > > > > reverted on master while we work on a fix.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > > > > <Mi...@comcast.com> wrote:
> > > > > >
> > > > > > In my opinion, in order to get to a cadence we are discussing we
> > > need to
> > > > > put a lot more work into the CI system. It has been failing
> > > consistently,
> > > > > doesn’t block PR approvals/merges and when it actually runs it
> does not
> > > > > test anything besides build and license headers.
> > > > > >
> > > > > > In the past couple of weeks we have had (off the top of my head):
> > > > > >
> > > > > > * Unit tests broken
> > > > > > * TO API tests broken
> > > > > > * Golang vendor issues
> > > > > > * TO Go build issues
> > > > > >
> > > > > > All were merged and broke master.
> > > > > >
> > > > > > Yes, having release branches sounds great and more aggressive
> cadence
> > > > > will minimize the amount of risk but I think that comes with the
> need
> > > to
> > > > > improve on our automated validation and testing.
> > > > > >
> > > > > > We have proven that we have not kept master in a good state and
> > > adding
> > > > > more releases (more branches) will make it even harder to keep that
> > > > > stability we are already fighting.
> > > > > >
> > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mi...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Yeah, I get it. No one company should be driving release
> > > > > schedules/scope.
> > > > > >
> > > > > > What I was really getting at is if ANY company has a recent
> > > version
> > > > > in a
> > > > > > test or prod-like enviro (meaning the version is being
> exercised
> > > and
> > > > > > thoroughly tested), then maybe we consider simply cutting a
> > > release
> > > > > from
> > > > > > that version. For example, imagine company X had this
> version in
> > > a
> > > > > > test/prod enviro:
> > > > > >
> > > > > > Master-10287.7e62d07 (
> > > > > >
> > > > >
> > >
> https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > > > > )
> > > > > >
> > > > > > Then I would be in favor of cutting the 4.0 release from
> > > > > > Master-10287.7e62d07 as we know it is proven to work based on
> > > > > company X's
> > > > > > actual use of it.
> > > > > >
> > > > > > The truth is (imo), we don't have the bandwidth to manually
> > > verify
> > > > > releases
> > > > > > that we are not using/have not used nor do we have the
> necessary
> > > > > automated
> > > > > > test coverage to verify these releases. So most of our
> releases
> > > are
> > > > > > significantly untested.
> > > > > >
> > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <
> ocket8888@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I'm really not a fan of allowing Comcast to dictate the
> release
> > > > > scope and
> > > > > > > schedule. If cherry-picking is too messy, then we can just
> cut
> > > new
> > > > > minor
> > > > > > > releases directly from master.
> > > > > > >
> > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > > mitchell852@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > I don't think it's as easy as cherry picking
> (backporting)
> > > > > certain
> > > > > > > features
> > > > > > > > into a release branch. I could be wrong but I really
> don't
> > > think
> > > > > it is.
> > > > > > > >
> > > > > > > > So what I'm hearing is that 4.0 gets cut from master and
> we
> > > go
> > > > > through
> > > > > > > with
> > > > > > > > our normal process of testing, validating, etc. At the
> same
> > > > > time, 4.1
> > > > > > > > branch is created from 4.0. Master moves on as normal.
> Let's
> > > say
> > > > > 25
> > > > > > > commits
> > > > > > > > come in to master during the next 6 weeks. During that 6
> > > weeks,
> > > > > we cherry
> > > > > > > > pick only the features that look interesting for 4.1?
> (the
> > > rest
> > > > > just stay
> > > > > > > > in master and will get picked up in 5.0?)
> > > > > > > >
> > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are
> roughly 6
> > > > > weeks apart
> > > > > > > and
> > > > > > > > have only "interesting" features so the releases are
> > > > > incrementally very
> > > > > > > > small and much easier to test/validate, however, those
> > > features
> > > > > might
> > > > > > > > depend on other commits. At some point, cherry picking
> > > becomes
> > > > > impossible
> > > > > > > > as the foundation of the code base has shifted so
> > > drastically.
> > > > > Cherry
> > > > > > > > picking will become very messy and introduce a high
> level of
> > > > > risk imo.
> > > > > > > >
> > > > > > > > Also, you'd have to manage the scope of these minor
> releases
> > > > > which we are
> > > > > > > > not very good at because the designated release manager
> has a
> > > > > full-time
> > > > > > > job
> > > > > > > > to attend to and now we have even more releases to
> > > > > > > > scope/test/validate/approve. I like the idea of faster
> > > releases
> > > > > but this
> > > > > > > > sounds like more work that nobody has the bandwidth for.
> > > > > > > >
> > > > > > > > Ok, so rather than shooting down an idea with no
> alternative
> > > > > proposal, I
> > > > > > > > propose this. At Comcast, we are trying to get to the
> point
> > > > > where we
> > > > > > > > release TC to prod from the head of master every few
> weeks.
> > > What
> > > > > if every
> > > > > > > > time Comcast does an internal upgrade, they note the
> commit
> > > hash
> > > > > and once
> > > > > > > > comcast is happy with the upgrade in their prod enviro,
> they
> > > > > propose a
> > > > > > > > release pinned to that commit hash? With this approach
> you'd
> > > end
> > > > > up with:
> > > > > > > >
> > > > > > > > 1. fast releases (hopefully every few weeks)
> > > > > > > > 2. well tested releases as it's in a real prod enviro
> > > > > > > > 3. little/no validation work for anyone
> > > > > > > >
> > > > > > > > Also, this doesn't have to be Comcast. If anyone else is
> out
> > > > > front at the
> > > > > > > > edge of master and running it in a trusted enviro, they
> could
> > > > > propose a
> > > > > > > > release as well.
> > > > > > > >
> > > > > > > > Jeremy
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > > > > david.neuman64@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have been thinking about how we can get better with
> our
> > > > > release
> > > > > > > > cadence.
> > > > > > > > > I feel like we have slowed to a crawl and not been as
> good
> > > as
> > > > > we should
> > > > > > > > > about how we release. Our last Major release was in
> March
> > > and
> > > > > we
> > > > > > > haven't
> > > > > > > > > had a real release since. Moving forward I would
> like to
> > > see
> > > > > us get
> > > > > > > on
> > > > > > > > a
> > > > > > > > > more consistent cadence and provide smaller releases
> that
> > > can
> > > > > be more
> > > > > > > > > easily digested by the community. It's my hope that
> we can
> > > > > get to a
> > > > > > > > > cadence where we are doing releases every 4 to 6 weeks,
> > > we are
> > > > > > > releasing
> > > > > > > > > in such a way that not all releases all required (e.g.
> if
> > > you
> > > > > are on
> > > > > > > 4.0
> > > > > > > > > you don't need 4.1, you can just install 4.2), and we
> are
> > > more
> > > > > > > deliberate
> > > > > > > > > about what we put into a release.
> > > > > > > > >
> > > > > > > > > I think we can accomplish this by using our release
> > > branches
> > > > > better.
> > > > > > > For
> > > > > > > > > each major release we will cut a release branch from
> master
> > > > > (e.g. 4.x)
> > > > > > > > and
> > > > > > > > > then we will use cherry-picking to add new features
> and bug
> > > > > fixes to
> > > > > > > that
> > > > > > > > > release. This means that if 4.0 has been released and
> you
> > > > > want to get
> > > > > > > > your
> > > > > > > > > feature/bug fix in 4.1, you will first submit your PR
> to
> > > > > master and
> > > > > > > then
> > > > > > > > > cheery pick your squash merged commit to the 4.x branch
> > > which
> > > > > we will
> > > > > > > use
> > > > > > > > > to create the 4.1 release. I think we should allow
> either
> > > the
> > > > > > > > contributor
> > > > > > > > > or the committer (who is merging the PR) to suggest if
> a
> > > > > feature goes
> > > > > > > > into
> > > > > > > > > the release branch, and if we disagree we can take it
> to
> > > the
> > > > > list. If
> > > > > > > we
> > > > > > > > > decide that every PR to master also goes into 4.1 then
> so
> > > be
> > > > > it, but at
> > > > > > > > > least we are making a conscious decision of what goes
> into
> > > the
> > > > > next
> > > > > > > > > release. This will allow us to not be so worried
> about
> > > what
> > > > > we are
> > > > > > > > > merging into master and how that will affect our next
> > > release.
> > > > > > > According
> > > > > > > > > to our 6 week cadence we will cut a new release from
> the
> > > > > current
> > > > > > > feature
> > > > > > > > > branch and put that up for a vote. These releases
> will be
> > > > > small and
> > > > > > > > > testable enough that we can get feedback quickly and
> move
> > > from
> > > > > RC to
> > > > > > > > > release in a short time. If a release has too many
> issues
> > > > > then we may
> > > > > > > > end
> > > > > > > > > up deciding to skip that minor release in favor of the
> next
> > > > > one,
> > > > > > > > hopefully
> > > > > > > > > this does not happen very often if at all.
> > > > > > > > >
> > > > > > > > > Once a breaking change is introduced to master which
> will
> > > > > require a new
> > > > > > > > > major release, we will create a new major release
> branch
> > > from
> > > > > master
> > > > > > > > (which
> > > > > > > > > will mean a new release manager). We will then repeat
> the
> > > > > same process
> > > > > > > > > with the new release branch, etc, etc.
> > > > > > > > >
> > > > > > > > > As for LTS we will provide support for the latest major
> > > > > release plus
> > > > > > > the
> > > > > > > > > one previous. So, once we release 4.0 we will support
> 4.0
> > > and
> > > > > 3.1.
> > > > > > > Any
> > > > > > > > > security issues that arise will -- if present -- be
> > > applied to
> > > > > each of
> > > > > > > > > these versions. Once 4.1 is released we will support
> 4.1
> > > and
> > > > > 3.1, etc,
> > > > > > > > > etc.
> > > > > > > > >
> > > > > > > > > Please let me know if you have any thoughts on this
> plan.
> > > If
> > > > > there are
> > > > > > > > no
> > > > > > > > > major objections I would like to try this with 4.x
> with the
> > > > > idea that
> > > > > > > we
> > > > > > > > > will adjust how we do things as necessary.
> > > > > > > > >
> > > > > > > > > We need to get better at releasing and something needs
> to
> > > > > change,
> > > > > > > > hopefully
> > > > > > > > > this can get us going in the right direction.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dave
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > TL;DR - I am proposing a 6 week release cycle using
> > > > > cherry-picks to
> > > > > > > our
> > > > > > > > > release branch to control what goes into a minor
> release.
> > > > > Major
> > > > > > > releases
> > > > > > > > > will be cut from master as necessary - such as if
> there is
> > > a
> > > > > breaking
> > > > > > > > > change that is introduced.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
>
Re: [EXTERNAL] Re: Release Process
Posted by Rawlin Peters <ra...@apache.org>.
> I'm +1 on this final summary. It infers that we are going to soon be adding a job that runs CI checks on Master. This is good. That's probably a subject for another thread. I do have one question about this approach. I think there is probably a scenario where we need to fix an issue on the 4.x branch itself. In this case I guess we would merge that commit back to master? And then RM would have to keep track of when its safe to fast-forward to the merged version of the fix?
My understanding is that we shouldn't ever have a reason to fix an
issue directly on the 4.x branch itself. When 4.x is ready for a
release, we will cut a new branch (e.g. 4.1.x) off of 4.x at that
point. If we find we absolutely need a fix for 4.1.x, then ideally we
should develop the fix on master then cherry-pick it into 4.1.x. I'm
not sure we'd also cherry-pick it into 4.x since then 4.x would no
longer be a snapshot of the most recent stable commit of master
(meaning we wouldn't be able to just "fast-forward" -- in git
terminology -- to the next most recent stable commit of master).
IMO unless an absolutely necessary fix is also super time-sensitive,
we should develop the fix on master and cherry-pick it into the 4.1.x
branch. Then all cherry-picks would be tracked the same way, without
also having to track hotfixes that need cherry-picked to master.
Whenever cherry-picks/hotfixes are used, the important thing is just
to make sure they get carried forward into the next release (either by
bringing them in naturally via moving forward on master, or
re-cherry-picking them). Again, these should only be things we deem
absolutely necessary for a release (show-stoppers).
- Rawlin