You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Benjamin Mahler <bm...@apache.org> on 2018/07/12 23:17:59 UTC
Re: Backport Policy
+user, I probably it would be good to hear from users as well.
Please see the original proposal as well as Alex's proposal and let us know
your thoughts.
To continue the discussion from where Alex left off:
> Other bugs and significant improvements, e.g., performance, may be back ported,
the release manager should ideally be the one who decides on this.
I'm a little puzzled by this, why is the release manager involved? As we
already document, backports occur when the bug is fixed, so this happens in
the steady state of development, not at release time. The release manager
only comes in at the time of the release itself, at which point all
backports have already happened and the release manager handles the release
process. Only blocker level issues can stop the release and while the
release manager has a strong say, we should generally agree on what
consists of a release blocking issue.
Just to clarify my workflow, I generally backport every bug fix I commit
that applies cleanly, right after I commit it to master (with the
exceptions I listed below).
On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com> wrote:
> I would like to back port as little as possible. I suggest the following
> criteria:
>
> * By default, regressions are back ported to existing release branches. A
> bug is considered a regression if the functionality is present in the
> previous minor or patch version and is not affected by the bug there.
>
> * Critical and blocker issues, e.g., a CVE, can be back ported.
>
> * Other bugs and significant improvements, e.g., performance, may be back
> ported, the release manager should ideally be the one who decides on this.
>
> On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org> wrote:
>
> > Ben, thanks for the clarification. I'm in agreement with the points you
> > made.
> >
> > Once we have consensus, would you mind updating the doc?
> >
> > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
> > wrote:
> >
> > > I realized recently that we aren't all on the same page with
> backporting.
> > > We currently only document the following:
> > >
> > > "Typically the fix for an issue that is affecting supported releases
> > lands
> > > on the master branch and is then backported to the release branch(es).
> In
> > > rare cases, the fix might directly go into a release branch without
> > landing
> > > on master (e.g., fix / issue is not applicable to master)." [1]
> > >
> > > This leaves room for interpretation about what lies outside of
> "typical".
> > > Here's the simplest way I can explain what I stick to, and I'd like to
> > hear
> > > what others have in mind:
> > >
> > > * By default, bug fixes at any level should be backported to existing
> > > release branches if it affects those releases. Especially important:
> > > crashes, bugs in non-experimental features.
> > >
> > > * Exceptional cases that can omit backporting: difficult to backport
> > fixes
> > > (especially if the bugs are deemed of low priority), bugs in
> experimental
> > > features.
> > >
> > > * Exceptional non-bug cases that can be backported: performance
> > > improvements.
> > >
> > > I realize that there is a ton of subtlety here (even in terms of which
> > > things are defined as bugs). But I hope we can lay down a policy that
> > gives
> > > everyone the right mindset for common cases and then discuss corner
> cases
> > > on-demand in the future.
> > >
> > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > >
> >
>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
Thanks Ben!
I wonder if the new guidelines belong in the "Committing" docs instead?
http://mesos.apache.org/documentation/latest/committing/
On Thu, Jul 26, 2018 at 2:18 PM, Benjamin Mahler <bm...@apache.org> wrote:
> Thanks to everyone who chimed in. To Gilbert's questions, I think that's
> outside the territory of what a policy or guidelines will cover, and will
> be up to the judgement of the committer.
>
> My initial search for "backport" was insufficient and I missed the
> following existing comment in a different section of versioning.md lays out
> a policy of only critical issues:
>
> <start quote>
> At any given time, 3 releases are supported: ... Support means fixing of
> critical issues that affect the release. Once an issue is deemed critical,
> it will be fixed in only those affected releases that are still supported.
> ...
>
> Which issues are considered critical?
>
> Security fixes
> Compatibility regressions
> Functional regressions
> Performance regressions
> Fixes for 3rd party integration (e.g., Docker remote API)
>
> Whether an issue is considered critical or not is sometimes subjective. In
> some cases it is obvious and sometimes it is fuzzy. Users should work with
> committers to figure out the criticality of an issue and get agreement and
> commitment for support.
> <end quote>
>
> From the discussion here, it sounds like we should add the guidelines
> similar to what Greg suggested above to clarify that there's some
> subjectivity here, as well as keeping some of the clear cases (like what
> the existing documentation does). I can send out a change sometime soon.
>
> On Wed, Jul 18, 2018 at 10:37 AM, Gilbert Song <gi...@mesosphere.io>
> wrote:
>
>> Thanks for clarifying the backporting policy, BenM!
>>
>> I totally agree with the changes proposed for the backporting policy, but
>> I realize two more scenarios that are more clear to me yet:
>>
>> - There are some bugs that are not fixable (due to legacy technical
>> decisions), and we end up with fixing the issue by a semantic/behavior
>> change in a new release. Do we expect this semantic/behavior change being
>> backported?
>> - There might be some bugs that root cause is unknown yet, but it did
>> impact on a couple releases. If we decide to add some commits for debugging
>> purpose (e.g., a new debugging endpoint, or more logging), should we also
>> allow these patches to be backported?
>>
>> For #2, I think we should do the backporting, but for #1, maybe more
>> discussion is needed since it relates to whether the user has to upgrade or
>> not.
>>
>> Cheers,
>> Gilbert
>>
>> On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
>>
>>> I don’t have a big stake in, however, one opinion is if a large
>>> commercial enterprise was using a specific release that is working the
>>> desire is often to only upgrade if necessary. Necessary can be for a
>>> number of reasons including new features; however if a new feature is not
>>> needed the compelling reason to upgrade is to fix a specific problem that
>>> is causing issues. Thus keeping a maintenance release stable is very
>>> important and reducing the chance for, while fixing one problem,
>>> introducing another.
>>>
>>> Often a clear classification of severity of the problem would dictate
>>> the need to make a change. (yes these can be subjective, but some guidance
>>> would be better than nothing).
>>>
>>> It might be good to give committers guidance on back porting things that
>>> have a high impact on improving a problem. Fixing a crashing bug, fixing a
>>> degenerative performance issue, etc, where these issues have no easy/viable
>>> work around. Nice to have fixes aren’t, always, worth updating to.
>>>
>>> There can be an argument to respond with a “then don’t upgrade” but if
>>> changing the release with “nice to have’s” and several point releases later
>>> when a critical bug is fixed then the org if forced to accept the risk of
>>> the nice to have’s.
>>>
>>> just an opinion.
>>> …larry
>>>
>>>
>>> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
>>> wrote:
>>>
>>> I just have a comment on a special case:
>>> If a fix for a flaky test is easy to backport,
>>> IMO we probably should backport it,
>>> otherwise if someone backports another critical fix in the future,
>>> it would take them extra effort to check all CI failures.
>>>
>>> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org>
>>> wrote:
>>>
>>>> I like how you summarized it Greg and I would vote for leaving the
>>>> decision
>>>> to the committer too. In addition to what others mentioned, I think
>>>> committer should've the responsibility because if things break in a
>>>> point
>>>> release (after it is released), it is the committer and contributor who
>>>> are
>>>> on the hook to triage and fix it and not the release manager.
>>>>
>>>> Having said that, if "during" the release process (i.e., cutting an RC)
>>>> these backports cause delays for a release manager in getting the
>>>> release
>>>> out (e.g., CI flakiness introduced due to backports), release manager
>>>> could
>>>> be the ultimate arbiter on whether such a backport should be reverted or
>>>> fixed by the committer/contributor. Hopefully such issues are caught
>>>> much
>>>> before a release process is started (e.g., CI running against release
>>>> branches).
>>>>
>>>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>>>
>>>> > Greg, I like your idea of adding a prescriptive "policy" when
>>>> evaluating
>>>> > whether a bug fix should be backported, and leave the decision to
>>>> committer
>>>> > (because they have the most context, and avoid a bottleneck in the
>>>> > process).
>>>> >
>>>> > - Jie
>>>> >
>>>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io>
>>>> wrote:
>>>> >
>>>> > > My impression is that we have two opposing schools of thought here:
>>>> > >
>>>> > > 1. Backport as little as possible, to avoid unforeseen
>>>> consequences
>>>> > > 2. Backport as much as proves practical, to eliminate bugs in
>>>> > > supported versions
>>>> > >
>>>> > > Do other people agree with this assessment?
>>>> > >
>>>> > > If so, how can we find common ground? One possible solution would
>>>> be to
>>>> > > leave the decision on backporting up to the committer, without
>>>> > specifying a
>>>> > > project-wide policy. This seems to be the status quo, and would
>>>> lead to
>>>> > > some variation across committers regarding what types of fixes are
>>>> > > backported. We could also choose to delegate the decision to the
>>>> release
>>>> > > manager; I favor leaving the decision with the committer, to
>>>> eliminate
>>>> > the
>>>> > > burden on release managers.
>>>> > >
>>>> > > Here's a thought: rather than defining a prescriptive "policy" that
>>>> we
>>>> > > expect committers to abide by, we could enumerate in the
>>>> documentation
>>>> > the
>>>> > > competing concerns that we expect committers to consider when making
>>>> > > decisions on backports. The committing docs could read something
>>>> like:
>>>> > >
>>>> > > "When bug fixes are committed to master, the committer should
>>>> evaluate
>>>> > the
>>>> > > fix to determine whether or not it should be backported to supported
>>>> > > versions. This is left to the committer, but they are expected to
>>>> weigh
>>>> > the
>>>> > > following concerns when making the decision:
>>>> > >
>>>> > > - Every backported change comes with a risk of unintended
>>>> > > consequences. The change should be carefully evaluated to ensure
>>>> that
>>>> > such
>>>> > > side-effects are highly unlikely.
>>>> > > - As the complexity of applying a backport increases due to merge
>>>> > > conflicts, the likelihood of unintended consequences also
>>>> increases.
>>>> > Bug
>>>> > > fixes which require extensive rebasing should only be backported
>>>> when
>>>> > the
>>>> > > bug is critical enough to warrant the risk.
>>>> > > - Users of supported versions benefit greatly from the
>>>> resolution of
>>>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>>>> > allayed
>>>> > > for a given bug fix, it should be backported."
>>>> > >
>>>> > >
>>>> > > Cheers,
>>>> > > Greg
>>>> > >
>>>> > >
>>>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <
>>>> alex@mesosphere.com>
>>>> > > wrote:
>>>> > >
>>>> > >> Back porting as little as possible is the ultimate goal for me. My
>>>> > >> reasons are closely aligned with what Andrew wrote above.
>>>> > >>
>>>> > >> If we agree on this strategy, the next question is how to enforce
>>>> it. My
>>>> > >> intuition is that committers will lean towards back porting their
>>>> > patches
>>>> > >> in arguable cases, because humans tend to overestimate the
>>>> importance of
>>>> > >> their personal work. Delegating the decision in such cases to a
>>>> release
>>>> > >> manager in my opinion will help us enforce the strategy of minimal
>>>> > number
>>>> > >> backports. As a bonus, the release manager will have a much better
>>>> > >> understanding of what's going on with the release, keyword: "more
>>>> > >> ownership".
>>>> > >>
>>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>>>> > >> andrew@schwartzmeyer.com> wrote:
>>>> > >>
>>>> > >>> I believe I fall somewhere between Alex and Ben.
>>>> > >>>
>>>> > >>> As for deciding what to backport or not, I lean toward Alex's
>>>> view of
>>>> > >>> backporting as little as possible (and agree with his criteria).
>>>> My
>>>> > >>> reasoning is that all changes can have unforeseen consequences,
>>>> which I
>>>> > >>> believe is something to be actively avoided in already released
>>>> > versions.
>>>> > >>> The reason for backporting patches to fix regressions is the same
>>>> as
>>>> > the
>>>> > >>> reason to avoid backporting as much as possible: keep behavior
>>>> > consistent
>>>> > >>> (and safe) within a release. With that as the goal of a branch in
>>>> > >>> maintenance mode, it makes sense to fix regressions, and make
>>>> > exceptions to
>>>> > >>> fix CVEs and other critical/blocking issues.
>>>> > >>>
>>>> > >>> As for who should decide what to backport, I lean toward Ben's
>>>> view of
>>>> > >>> the burden being on the committer. I don't think we should add
>>>> more
>>>> > work
>>>> > >>> for release managers, and I think the committer/shepherd
>>>> obviously has
>>>> > the
>>>> > >>> most understanding of the context around changes proposed for
>>>> backport.
>>>> > >>>
>>>> > >>> Here's an example of a recent bugfix which I backported:
>>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>>> > >>>
>>>> > >>> While normally I believe this change falls under "avoid due to
>>>> > >>> unforeseen consequences," I made an exception as the bug was old,
>>>> circa
>>>> > >>> 2015, (indicating it had been an issue for others), and was
>>>> causing
>>>> > >>> recurring failures in testing. The fix itself was very small,
>>>> meaning
>>>> > it
>>>> > >>> was easier to evaluate for possible side effects, so I felt a
>>>> little
>>>> > safer
>>>> > >>> in that regard. The effect of not having the fix was a fatal and
>>>> > undesired
>>>> > >>> crash, which furthermore left troublesome side effects on the
>>>> system
>>>> > (you
>>>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>>>> > (DC/OS)
>>>> > >>> wanted it in their next bump, which necessitated backporting to
>>>> the
>>>> > release
>>>> > >>> they were pulling in.
>>>> > >>>
>>>> > >>> I think in general we should backport only as necessary, and
>>>> leave it
>>>> > on
>>>> > >>> the committers to decide if backporting a particular change is
>>>> > necessary.
>>>> > >>>
>>>> > >>>
>>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>>> > >>>
>>>> > >>>> This is exactly where our views differ, Ben : )
>>>> > >>>>
>>>> > >>>> Ideally, I would like a release manager to have more ownership
>>>> and
>>>> > less
>>>> > >>>> manual work. In my imagination, a release manager has more power
>>>> and
>>>> > >>>> control about dates, features, backports and everything that is
>>>> > related
>>>> > >>>> to
>>>> > >>>> "their" branch. I would also like us to back port as little as
>>>> > >>>> possible, to
>>>> > >>>> simplify testing and releasing patch versions.
>>>> > >>>>
>>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>>>> bmahler@apache.org>
>>>> > >>>> wrote:
>>>> > >>>>
>>>> > >>>> +user, I probably it would be good to hear from users as well.
>>>> > >>>>>
>>>> > >>>>> Please see the original proposal as well as Alex's proposal and
>>>> let
>>>> > us
>>>> > >>>>> know
>>>> > >>>>> your thoughts.
>>>> > >>>>>
>>>> > >>>>> To continue the discussion from where Alex left off:
>>>> > >>>>>
>>>> > >>>>> > Other bugs and significant improvements, e.g., performance,
>>>> may be
>>>> > >>>>> back
>>>> > >>>>> ported,
>>>> > >>>>> the release manager should ideally be the one who decides on
>>>> this.
>>>> > >>>>>
>>>> > >>>>> I'm a little puzzled by this, why is the release manager
>>>> involved? As
>>>> > >>>>> we
>>>> > >>>>> already document, backports occur when the bug is fixed, so this
>>>> > >>>>> happens in
>>>> > >>>>> the steady state of development, not at release time. The
>>>> release
>>>> > >>>>> manager
>>>> > >>>>> only comes in at the time of the release itself, at which point
>>>> all
>>>> > >>>>> backports have already happened and the release manager handles
>>>> the
>>>> > >>>>> release
>>>> > >>>>> process. Only blocker level issues can stop the release and
>>>> while the
>>>> > >>>>> release manager has a strong say, we should generally agree on
>>>> what
>>>> > >>>>> consists of a release blocking issue.
>>>> > >>>>>
>>>> > >>>>> Just to clarify my workflow, I generally backport every bug fix
>>>> I
>>>> > >>>>> commit
>>>> > >>>>> that applies cleanly, right after I commit it to master (with
>>>> the
>>>> > >>>>> exceptions I listed below).
>>>> > >>>>>
>>>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>>>> alex@mesosphere.com
>>>> > >
>>>> > >>>>> wrote:
>>>> > >>>>>
>>>> > >>>>> > I would like to back port as little as possible. I suggest the
>>>> > >>>>> following
>>>> > >>>>> > criteria:
>>>> > >>>>> >
>>>> > >>>>> > * By default, regressions are back ported to existing release
>>>> > >>>>> branches. A
>>>> > >>>>> > bug is considered a regression if the functionality is
>>>> present in
>>>> > the
>>>> > >>>>> > previous minor or patch version and is not affected by the bug
>>>> > there.
>>>> > >>>>> >
>>>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back
>>>> ported.
>>>> > >>>>> >
>>>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>>>> may
>>>> > be
>>>> > >>>>> back
>>>> > >>>>> > ported, the release manager should ideally be the one who
>>>> decides
>>>> > on
>>>> > >>>>> this.
>>>> > >>>>> >
>>>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>>>> vinodkone@apache.org
>>>> > >
>>>> > >>>>> wrote:
>>>> > >>>>> >
>>>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>>> > >>>>> points you
>>>> > >>>>> > > made.
>>>> > >>>>> > >
>>>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>>>> > >>>>> > >
>>>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>>> > >>>>> bmahler@apache.org>
>>>> > >>>>> > > wrote:
>>>> > >>>>> > >
>>>> > >>>>> > > > I realized recently that we aren't all on the same page
>>>> with
>>>> > >>>>> > backporting.
>>>> > >>>>> > > > We currently only document the following:
>>>> > >>>>> > > >
>>>> > >>>>> > > > "Typically the fix for an issue that is affecting
>>>> supported
>>>> > >>>>> releases
>>>> > >>>>> > > lands
>>>> > >>>>> > > > on the master branch and is then backported to the release
>>>> > >>>>> branch(es).
>>>> > >>>>> > In
>>>> > >>>>> > > > rare cases, the fix might directly go into a release
>>>> branch
>>>> > >>>>> without
>>>> > >>>>> > > landing
>>>> > >>>>> > > > on master (e.g., fix / issue is not applicable to
>>>> master)." [1]
>>>> > >>>>> > > >
>>>> > >>>>> > > > This leaves room for interpretation about what lies
>>>> outside of
>>>> > >>>>> > "typical".
>>>> > >>>>> > > > Here's the simplest way I can explain what I stick to,
>>>> and I'd
>>>> > >>>>> like
>>>> > >>>>> to
>>>> > >>>>> > > hear
>>>> > >>>>> > > > what others have in mind:
>>>> > >>>>> > > >
>>>> > >>>>> > > > * By default, bug fixes at any level should be backported
>>>> to
>>>> > >>>>> existing
>>>> > >>>>> > > > release branches if it affects those releases. Especially
>>>> > >>>>> important:
>>>> > >>>>> > > > crashes, bugs in non-experimental features.
>>>> > >>>>> > > >
>>>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult
>>>> to
>>>> > >>>>> backport
>>>> > >>>>> > > fixes
>>>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs
>>>> in
>>>> > >>>>> > experimental
>>>> > >>>>> > > > features.
>>>> > >>>>> > > >
>>>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>>>> performance
>>>> > >>>>> > > > improvements.
>>>> > >>>>> > > >
>>>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>>>> terms
>>>> > of
>>>> > >>>>> which
>>>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>>>> > policy
>>>> > >>>>> that
>>>> > >>>>> > > gives
>>>> > >>>>> > > > everyone the right mindset for common cases and then
>>>> discuss
>>>> > >>>>> corner
>>>> > >>>>> > cases
>>>> > >>>>> > > > on-demand in the future.
>>>> > >>>>> > > >
>>>> > >>>>> > > > [1] http://mesos.apache.org/docume
>>>> ntation/latest/versioning/
>>>> > >>>>> > > >
>>>> > >>>>> > >
>>>> > >>>>> >
>>>> > >>>>>
>>>> > >>>>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
Thanks Ben!
I wonder if the new guidelines belong in the "Committing" docs instead?
http://mesos.apache.org/documentation/latest/committing/
On Thu, Jul 26, 2018 at 2:18 PM, Benjamin Mahler <bm...@apache.org> wrote:
> Thanks to everyone who chimed in. To Gilbert's questions, I think that's
> outside the territory of what a policy or guidelines will cover, and will
> be up to the judgement of the committer.
>
> My initial search for "backport" was insufficient and I missed the
> following existing comment in a different section of versioning.md lays out
> a policy of only critical issues:
>
> <start quote>
> At any given time, 3 releases are supported: ... Support means fixing of
> critical issues that affect the release. Once an issue is deemed critical,
> it will be fixed in only those affected releases that are still supported.
> ...
>
> Which issues are considered critical?
>
> Security fixes
> Compatibility regressions
> Functional regressions
> Performance regressions
> Fixes for 3rd party integration (e.g., Docker remote API)
>
> Whether an issue is considered critical or not is sometimes subjective. In
> some cases it is obvious and sometimes it is fuzzy. Users should work with
> committers to figure out the criticality of an issue and get agreement and
> commitment for support.
> <end quote>
>
> From the discussion here, it sounds like we should add the guidelines
> similar to what Greg suggested above to clarify that there's some
> subjectivity here, as well as keeping some of the clear cases (like what
> the existing documentation does). I can send out a change sometime soon.
>
> On Wed, Jul 18, 2018 at 10:37 AM, Gilbert Song <gi...@mesosphere.io>
> wrote:
>
>> Thanks for clarifying the backporting policy, BenM!
>>
>> I totally agree with the changes proposed for the backporting policy, but
>> I realize two more scenarios that are more clear to me yet:
>>
>> - There are some bugs that are not fixable (due to legacy technical
>> decisions), and we end up with fixing the issue by a semantic/behavior
>> change in a new release. Do we expect this semantic/behavior change being
>> backported?
>> - There might be some bugs that root cause is unknown yet, but it did
>> impact on a couple releases. If we decide to add some commits for debugging
>> purpose (e.g., a new debugging endpoint, or more logging), should we also
>> allow these patches to be backported?
>>
>> For #2, I think we should do the backporting, but for #1, maybe more
>> discussion is needed since it relates to whether the user has to upgrade or
>> not.
>>
>> Cheers,
>> Gilbert
>>
>> On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
>>
>>> I don’t have a big stake in, however, one opinion is if a large
>>> commercial enterprise was using a specific release that is working the
>>> desire is often to only upgrade if necessary. Necessary can be for a
>>> number of reasons including new features; however if a new feature is not
>>> needed the compelling reason to upgrade is to fix a specific problem that
>>> is causing issues. Thus keeping a maintenance release stable is very
>>> important and reducing the chance for, while fixing one problem,
>>> introducing another.
>>>
>>> Often a clear classification of severity of the problem would dictate
>>> the need to make a change. (yes these can be subjective, but some guidance
>>> would be better than nothing).
>>>
>>> It might be good to give committers guidance on back porting things that
>>> have a high impact on improving a problem. Fixing a crashing bug, fixing a
>>> degenerative performance issue, etc, where these issues have no easy/viable
>>> work around. Nice to have fixes aren’t, always, worth updating to.
>>>
>>> There can be an argument to respond with a “then don’t upgrade” but if
>>> changing the release with “nice to have’s” and several point releases later
>>> when a critical bug is fixed then the org if forced to accept the risk of
>>> the nice to have’s.
>>>
>>> just an opinion.
>>> …larry
>>>
>>>
>>> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
>>> wrote:
>>>
>>> I just have a comment on a special case:
>>> If a fix for a flaky test is easy to backport,
>>> IMO we probably should backport it,
>>> otherwise if someone backports another critical fix in the future,
>>> it would take them extra effort to check all CI failures.
>>>
>>> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org>
>>> wrote:
>>>
>>>> I like how you summarized it Greg and I would vote for leaving the
>>>> decision
>>>> to the committer too. In addition to what others mentioned, I think
>>>> committer should've the responsibility because if things break in a
>>>> point
>>>> release (after it is released), it is the committer and contributor who
>>>> are
>>>> on the hook to triage and fix it and not the release manager.
>>>>
>>>> Having said that, if "during" the release process (i.e., cutting an RC)
>>>> these backports cause delays for a release manager in getting the
>>>> release
>>>> out (e.g., CI flakiness introduced due to backports), release manager
>>>> could
>>>> be the ultimate arbiter on whether such a backport should be reverted or
>>>> fixed by the committer/contributor. Hopefully such issues are caught
>>>> much
>>>> before a release process is started (e.g., CI running against release
>>>> branches).
>>>>
>>>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>>>
>>>> > Greg, I like your idea of adding a prescriptive "policy" when
>>>> evaluating
>>>> > whether a bug fix should be backported, and leave the decision to
>>>> committer
>>>> > (because they have the most context, and avoid a bottleneck in the
>>>> > process).
>>>> >
>>>> > - Jie
>>>> >
>>>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io>
>>>> wrote:
>>>> >
>>>> > > My impression is that we have two opposing schools of thought here:
>>>> > >
>>>> > > 1. Backport as little as possible, to avoid unforeseen
>>>> consequences
>>>> > > 2. Backport as much as proves practical, to eliminate bugs in
>>>> > > supported versions
>>>> > >
>>>> > > Do other people agree with this assessment?
>>>> > >
>>>> > > If so, how can we find common ground? One possible solution would
>>>> be to
>>>> > > leave the decision on backporting up to the committer, without
>>>> > specifying a
>>>> > > project-wide policy. This seems to be the status quo, and would
>>>> lead to
>>>> > > some variation across committers regarding what types of fixes are
>>>> > > backported. We could also choose to delegate the decision to the
>>>> release
>>>> > > manager; I favor leaving the decision with the committer, to
>>>> eliminate
>>>> > the
>>>> > > burden on release managers.
>>>> > >
>>>> > > Here's a thought: rather than defining a prescriptive "policy" that
>>>> we
>>>> > > expect committers to abide by, we could enumerate in the
>>>> documentation
>>>> > the
>>>> > > competing concerns that we expect committers to consider when making
>>>> > > decisions on backports. The committing docs could read something
>>>> like:
>>>> > >
>>>> > > "When bug fixes are committed to master, the committer should
>>>> evaluate
>>>> > the
>>>> > > fix to determine whether or not it should be backported to supported
>>>> > > versions. This is left to the committer, but they are expected to
>>>> weigh
>>>> > the
>>>> > > following concerns when making the decision:
>>>> > >
>>>> > > - Every backported change comes with a risk of unintended
>>>> > > consequences. The change should be carefully evaluated to ensure
>>>> that
>>>> > such
>>>> > > side-effects are highly unlikely.
>>>> > > - As the complexity of applying a backport increases due to merge
>>>> > > conflicts, the likelihood of unintended consequences also
>>>> increases.
>>>> > Bug
>>>> > > fixes which require extensive rebasing should only be backported
>>>> when
>>>> > the
>>>> > > bug is critical enough to warrant the risk.
>>>> > > - Users of supported versions benefit greatly from the
>>>> resolution of
>>>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>>>> > allayed
>>>> > > for a given bug fix, it should be backported."
>>>> > >
>>>> > >
>>>> > > Cheers,
>>>> > > Greg
>>>> > >
>>>> > >
>>>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <
>>>> alex@mesosphere.com>
>>>> > > wrote:
>>>> > >
>>>> > >> Back porting as little as possible is the ultimate goal for me. My
>>>> > >> reasons are closely aligned with what Andrew wrote above.
>>>> > >>
>>>> > >> If we agree on this strategy, the next question is how to enforce
>>>> it. My
>>>> > >> intuition is that committers will lean towards back porting their
>>>> > patches
>>>> > >> in arguable cases, because humans tend to overestimate the
>>>> importance of
>>>> > >> their personal work. Delegating the decision in such cases to a
>>>> release
>>>> > >> manager in my opinion will help us enforce the strategy of minimal
>>>> > number
>>>> > >> backports. As a bonus, the release manager will have a much better
>>>> > >> understanding of what's going on with the release, keyword: "more
>>>> > >> ownership".
>>>> > >>
>>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>>>> > >> andrew@schwartzmeyer.com> wrote:
>>>> > >>
>>>> > >>> I believe I fall somewhere between Alex and Ben.
>>>> > >>>
>>>> > >>> As for deciding what to backport or not, I lean toward Alex's
>>>> view of
>>>> > >>> backporting as little as possible (and agree with his criteria).
>>>> My
>>>> > >>> reasoning is that all changes can have unforeseen consequences,
>>>> which I
>>>> > >>> believe is something to be actively avoided in already released
>>>> > versions.
>>>> > >>> The reason for backporting patches to fix regressions is the same
>>>> as
>>>> > the
>>>> > >>> reason to avoid backporting as much as possible: keep behavior
>>>> > consistent
>>>> > >>> (and safe) within a release. With that as the goal of a branch in
>>>> > >>> maintenance mode, it makes sense to fix regressions, and make
>>>> > exceptions to
>>>> > >>> fix CVEs and other critical/blocking issues.
>>>> > >>>
>>>> > >>> As for who should decide what to backport, I lean toward Ben's
>>>> view of
>>>> > >>> the burden being on the committer. I don't think we should add
>>>> more
>>>> > work
>>>> > >>> for release managers, and I think the committer/shepherd
>>>> obviously has
>>>> > the
>>>> > >>> most understanding of the context around changes proposed for
>>>> backport.
>>>> > >>>
>>>> > >>> Here's an example of a recent bugfix which I backported:
>>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>>> > >>>
>>>> > >>> While normally I believe this change falls under "avoid due to
>>>> > >>> unforeseen consequences," I made an exception as the bug was old,
>>>> circa
>>>> > >>> 2015, (indicating it had been an issue for others), and was
>>>> causing
>>>> > >>> recurring failures in testing. The fix itself was very small,
>>>> meaning
>>>> > it
>>>> > >>> was easier to evaluate for possible side effects, so I felt a
>>>> little
>>>> > safer
>>>> > >>> in that regard. The effect of not having the fix was a fatal and
>>>> > undesired
>>>> > >>> crash, which furthermore left troublesome side effects on the
>>>> system
>>>> > (you
>>>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>>>> > (DC/OS)
>>>> > >>> wanted it in their next bump, which necessitated backporting to
>>>> the
>>>> > release
>>>> > >>> they were pulling in.
>>>> > >>>
>>>> > >>> I think in general we should backport only as necessary, and
>>>> leave it
>>>> > on
>>>> > >>> the committers to decide if backporting a particular change is
>>>> > necessary.
>>>> > >>>
>>>> > >>>
>>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>>> > >>>
>>>> > >>>> This is exactly where our views differ, Ben : )
>>>> > >>>>
>>>> > >>>> Ideally, I would like a release manager to have more ownership
>>>> and
>>>> > less
>>>> > >>>> manual work. In my imagination, a release manager has more power
>>>> and
>>>> > >>>> control about dates, features, backports and everything that is
>>>> > related
>>>> > >>>> to
>>>> > >>>> "their" branch. I would also like us to back port as little as
>>>> > >>>> possible, to
>>>> > >>>> simplify testing and releasing patch versions.
>>>> > >>>>
>>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>>>> bmahler@apache.org>
>>>> > >>>> wrote:
>>>> > >>>>
>>>> > >>>> +user, I probably it would be good to hear from users as well.
>>>> > >>>>>
>>>> > >>>>> Please see the original proposal as well as Alex's proposal and
>>>> let
>>>> > us
>>>> > >>>>> know
>>>> > >>>>> your thoughts.
>>>> > >>>>>
>>>> > >>>>> To continue the discussion from where Alex left off:
>>>> > >>>>>
>>>> > >>>>> > Other bugs and significant improvements, e.g., performance,
>>>> may be
>>>> > >>>>> back
>>>> > >>>>> ported,
>>>> > >>>>> the release manager should ideally be the one who decides on
>>>> this.
>>>> > >>>>>
>>>> > >>>>> I'm a little puzzled by this, why is the release manager
>>>> involved? As
>>>> > >>>>> we
>>>> > >>>>> already document, backports occur when the bug is fixed, so this
>>>> > >>>>> happens in
>>>> > >>>>> the steady state of development, not at release time. The
>>>> release
>>>> > >>>>> manager
>>>> > >>>>> only comes in at the time of the release itself, at which point
>>>> all
>>>> > >>>>> backports have already happened and the release manager handles
>>>> the
>>>> > >>>>> release
>>>> > >>>>> process. Only blocker level issues can stop the release and
>>>> while the
>>>> > >>>>> release manager has a strong say, we should generally agree on
>>>> what
>>>> > >>>>> consists of a release blocking issue.
>>>> > >>>>>
>>>> > >>>>> Just to clarify my workflow, I generally backport every bug fix
>>>> I
>>>> > >>>>> commit
>>>> > >>>>> that applies cleanly, right after I commit it to master (with
>>>> the
>>>> > >>>>> exceptions I listed below).
>>>> > >>>>>
>>>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>>>> alex@mesosphere.com
>>>> > >
>>>> > >>>>> wrote:
>>>> > >>>>>
>>>> > >>>>> > I would like to back port as little as possible. I suggest the
>>>> > >>>>> following
>>>> > >>>>> > criteria:
>>>> > >>>>> >
>>>> > >>>>> > * By default, regressions are back ported to existing release
>>>> > >>>>> branches. A
>>>> > >>>>> > bug is considered a regression if the functionality is
>>>> present in
>>>> > the
>>>> > >>>>> > previous minor or patch version and is not affected by the bug
>>>> > there.
>>>> > >>>>> >
>>>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back
>>>> ported.
>>>> > >>>>> >
>>>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>>>> may
>>>> > be
>>>> > >>>>> back
>>>> > >>>>> > ported, the release manager should ideally be the one who
>>>> decides
>>>> > on
>>>> > >>>>> this.
>>>> > >>>>> >
>>>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>>>> vinodkone@apache.org
>>>> > >
>>>> > >>>>> wrote:
>>>> > >>>>> >
>>>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>>> > >>>>> points you
>>>> > >>>>> > > made.
>>>> > >>>>> > >
>>>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>>>> > >>>>> > >
>>>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>>> > >>>>> bmahler@apache.org>
>>>> > >>>>> > > wrote:
>>>> > >>>>> > >
>>>> > >>>>> > > > I realized recently that we aren't all on the same page
>>>> with
>>>> > >>>>> > backporting.
>>>> > >>>>> > > > We currently only document the following:
>>>> > >>>>> > > >
>>>> > >>>>> > > > "Typically the fix for an issue that is affecting
>>>> supported
>>>> > >>>>> releases
>>>> > >>>>> > > lands
>>>> > >>>>> > > > on the master branch and is then backported to the release
>>>> > >>>>> branch(es).
>>>> > >>>>> > In
>>>> > >>>>> > > > rare cases, the fix might directly go into a release
>>>> branch
>>>> > >>>>> without
>>>> > >>>>> > > landing
>>>> > >>>>> > > > on master (e.g., fix / issue is not applicable to
>>>> master)." [1]
>>>> > >>>>> > > >
>>>> > >>>>> > > > This leaves room for interpretation about what lies
>>>> outside of
>>>> > >>>>> > "typical".
>>>> > >>>>> > > > Here's the simplest way I can explain what I stick to,
>>>> and I'd
>>>> > >>>>> like
>>>> > >>>>> to
>>>> > >>>>> > > hear
>>>> > >>>>> > > > what others have in mind:
>>>> > >>>>> > > >
>>>> > >>>>> > > > * By default, bug fixes at any level should be backported
>>>> to
>>>> > >>>>> existing
>>>> > >>>>> > > > release branches if it affects those releases. Especially
>>>> > >>>>> important:
>>>> > >>>>> > > > crashes, bugs in non-experimental features.
>>>> > >>>>> > > >
>>>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult
>>>> to
>>>> > >>>>> backport
>>>> > >>>>> > > fixes
>>>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs
>>>> in
>>>> > >>>>> > experimental
>>>> > >>>>> > > > features.
>>>> > >>>>> > > >
>>>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>>>> performance
>>>> > >>>>> > > > improvements.
>>>> > >>>>> > > >
>>>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>>>> terms
>>>> > of
>>>> > >>>>> which
>>>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>>>> > policy
>>>> > >>>>> that
>>>> > >>>>> > > gives
>>>> > >>>>> > > > everyone the right mindset for common cases and then
>>>> discuss
>>>> > >>>>> corner
>>>> > >>>>> > cases
>>>> > >>>>> > > > on-demand in the future.
>>>> > >>>>> > > >
>>>> > >>>>> > > > [1] http://mesos.apache.org/docume
>>>> ntation/latest/versioning/
>>>> > >>>>> > > >
>>>> > >>>>> > >
>>>> > >>>>> >
>>>> > >>>>>
>>>> > >>>>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Backport Policy
Posted by Benjamin Mahler <bm...@apache.org>.
Thanks to everyone who chimed in. To Gilbert's questions, I think that's
outside the territory of what a policy or guidelines will cover, and will
be up to the judgement of the committer.
My initial search for "backport" was insufficient and I missed the
following existing comment in a different section of versioning.md lays out
a policy of only critical issues:
<start quote>
At any given time, 3 releases are supported: ... Support means fixing of
critical issues that affect the release. Once an issue is deemed critical,
it will be fixed in only those affected releases that are still supported.
...
Which issues are considered critical?
Security fixes
Compatibility regressions
Functional regressions
Performance regressions
Fixes for 3rd party integration (e.g., Docker remote API)
Whether an issue is considered critical or not is sometimes subjective. In
some cases it is obvious and sometimes it is fuzzy. Users should work with
committers to figure out the criticality of an issue and get agreement and
commitment for support.
<end quote>
From the discussion here, it sounds like we should add the guidelines
similar to what Greg suggested above to clarify that there's some
subjectivity here, as well as keeping some of the clear cases (like what
the existing documentation does). I can send out a change sometime soon.
On Wed, Jul 18, 2018 at 10:37 AM, Gilbert Song <gi...@mesosphere.io>
wrote:
> Thanks for clarifying the backporting policy, BenM!
>
> I totally agree with the changes proposed for the backporting policy, but
> I realize two more scenarios that are more clear to me yet:
>
> - There are some bugs that are not fixable (due to legacy technical
> decisions), and we end up with fixing the issue by a semantic/behavior
> change in a new release. Do we expect this semantic/behavior change being
> backported?
> - There might be some bugs that root cause is unknown yet, but it did
> impact on a couple releases. If we decide to add some commits for debugging
> purpose (e.g., a new debugging endpoint, or more logging), should we also
> allow these patches to be backported?
>
> For #2, I think we should do the backporting, but for #1, maybe more
> discussion is needed since it relates to whether the user has to upgrade or
> not.
>
> Cheers,
> Gilbert
>
> On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
>
>> I don’t have a big stake in, however, one opinion is if a large
>> commercial enterprise was using a specific release that is working the
>> desire is often to only upgrade if necessary. Necessary can be for a
>> number of reasons including new features; however if a new feature is not
>> needed the compelling reason to upgrade is to fix a specific problem that
>> is causing issues. Thus keeping a maintenance release stable is very
>> important and reducing the chance for, while fixing one problem,
>> introducing another.
>>
>> Often a clear classification of severity of the problem would dictate the
>> need to make a change. (yes these can be subjective, but some guidance
>> would be better than nothing).
>>
>> It might be good to give committers guidance on back porting things that
>> have a high impact on improving a problem. Fixing a crashing bug, fixing a
>> degenerative performance issue, etc, where these issues have no easy/viable
>> work around. Nice to have fixes aren’t, always, worth updating to.
>>
>> There can be an argument to respond with a “then don’t upgrade” but if
>> changing the release with “nice to have’s” and several point releases later
>> when a critical bug is fixed then the org if forced to accept the risk of
>> the nice to have’s.
>>
>> just an opinion.
>> …larry
>>
>>
>> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
>> wrote:
>>
>> I just have a comment on a special case:
>> If a fix for a flaky test is easy to backport,
>> IMO we probably should backport it,
>> otherwise if someone backports another critical fix in the future,
>> it would take them extra effort to check all CI failures.
>>
>> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
>>
>>> I like how you summarized it Greg and I would vote for leaving the
>>> decision
>>> to the committer too. In addition to what others mentioned, I think
>>> committer should've the responsibility because if things break in a point
>>> release (after it is released), it is the committer and contributor who
>>> are
>>> on the hook to triage and fix it and not the release manager.
>>>
>>> Having said that, if "during" the release process (i.e., cutting an RC)
>>> these backports cause delays for a release manager in getting the release
>>> out (e.g., CI flakiness introduced due to backports), release manager
>>> could
>>> be the ultimate arbiter on whether such a backport should be reverted or
>>> fixed by the committer/contributor. Hopefully such issues are caught much
>>> before a release process is started (e.g., CI running against release
>>> branches).
>>>
>>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>>
>>> > Greg, I like your idea of adding a prescriptive "policy" when
>>> evaluating
>>> > whether a bug fix should be backported, and leave the decision to
>>> committer
>>> > (because they have the most context, and avoid a bottleneck in the
>>> > process).
>>> >
>>> > - Jie
>>> >
>>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io>
>>> wrote:
>>> >
>>> > > My impression is that we have two opposing schools of thought here:
>>> > >
>>> > > 1. Backport as little as possible, to avoid unforeseen
>>> consequences
>>> > > 2. Backport as much as proves practical, to eliminate bugs in
>>> > > supported versions
>>> > >
>>> > > Do other people agree with this assessment?
>>> > >
>>> > > If so, how can we find common ground? One possible solution would be
>>> to
>>> > > leave the decision on backporting up to the committer, without
>>> > specifying a
>>> > > project-wide policy. This seems to be the status quo, and would lead
>>> to
>>> > > some variation across committers regarding what types of fixes are
>>> > > backported. We could also choose to delegate the decision to the
>>> release
>>> > > manager; I favor leaving the decision with the committer, to
>>> eliminate
>>> > the
>>> > > burden on release managers.
>>> > >
>>> > > Here's a thought: rather than defining a prescriptive "policy" that
>>> we
>>> > > expect committers to abide by, we could enumerate in the
>>> documentation
>>> > the
>>> > > competing concerns that we expect committers to consider when making
>>> > > decisions on backports. The committing docs could read something
>>> like:
>>> > >
>>> > > "When bug fixes are committed to master, the committer should
>>> evaluate
>>> > the
>>> > > fix to determine whether or not it should be backported to supported
>>> > > versions. This is left to the committer, but they are expected to
>>> weigh
>>> > the
>>> > > following concerns when making the decision:
>>> > >
>>> > > - Every backported change comes with a risk of unintended
>>> > > consequences. The change should be carefully evaluated to ensure
>>> that
>>> > such
>>> > > side-effects are highly unlikely.
>>> > > - As the complexity of applying a backport increases due to merge
>>> > > conflicts, the likelihood of unintended consequences also
>>> increases.
>>> > Bug
>>> > > fixes which require extensive rebasing should only be backported
>>> when
>>> > the
>>> > > bug is critical enough to warrant the risk.
>>> > > - Users of supported versions benefit greatly from the resolution
>>> of
>>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>>> > allayed
>>> > > for a given bug fix, it should be backported."
>>> > >
>>> > >
>>> > > Cheers,
>>> > > Greg
>>> > >
>>> > >
>>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <alex@mesosphere.com
>>> >
>>> > > wrote:
>>> > >
>>> > >> Back porting as little as possible is the ultimate goal for me. My
>>> > >> reasons are closely aligned with what Andrew wrote above.
>>> > >>
>>> > >> If we agree on this strategy, the next question is how to enforce
>>> it. My
>>> > >> intuition is that committers will lean towards back porting their
>>> > patches
>>> > >> in arguable cases, because humans tend to overestimate the
>>> importance of
>>> > >> their personal work. Delegating the decision in such cases to a
>>> release
>>> > >> manager in my opinion will help us enforce the strategy of minimal
>>> > number
>>> > >> backports. As a bonus, the release manager will have a much better
>>> > >> understanding of what's going on with the release, keyword: "more
>>> > >> ownership".
>>> > >>
>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>>> > >> andrew@schwartzmeyer.com> wrote:
>>> > >>
>>> > >>> I believe I fall somewhere between Alex and Ben.
>>> > >>>
>>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>>> of
>>> > >>> backporting as little as possible (and agree with his criteria). My
>>> > >>> reasoning is that all changes can have unforeseen consequences,
>>> which I
>>> > >>> believe is something to be actively avoided in already released
>>> > versions.
>>> > >>> The reason for backporting patches to fix regressions is the same
>>> as
>>> > the
>>> > >>> reason to avoid backporting as much as possible: keep behavior
>>> > consistent
>>> > >>> (and safe) within a release. With that as the goal of a branch in
>>> > >>> maintenance mode, it makes sense to fix regressions, and make
>>> > exceptions to
>>> > >>> fix CVEs and other critical/blocking issues.
>>> > >>>
>>> > >>> As for who should decide what to backport, I lean toward Ben's
>>> view of
>>> > >>> the burden being on the committer. I don't think we should add more
>>> > work
>>> > >>> for release managers, and I think the committer/shepherd obviously
>>> has
>>> > the
>>> > >>> most understanding of the context around changes proposed for
>>> backport.
>>> > >>>
>>> > >>> Here's an example of a recent bugfix which I backported:
>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>> > >>>
>>> > >>> While normally I believe this change falls under "avoid due to
>>> > >>> unforeseen consequences," I made an exception as the bug was old,
>>> circa
>>> > >>> 2015, (indicating it had been an issue for others), and was causing
>>> > >>> recurring failures in testing. The fix itself was very small,
>>> meaning
>>> > it
>>> > >>> was easier to evaluate for possible side effects, so I felt a
>>> little
>>> > safer
>>> > >>> in that regard. The effect of not having the fix was a fatal and
>>> > undesired
>>> > >>> crash, which furthermore left troublesome side effects on the
>>> system
>>> > (you
>>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>>> > (DC/OS)
>>> > >>> wanted it in their next bump, which necessitated backporting to the
>>> > release
>>> > >>> they were pulling in.
>>> > >>>
>>> > >>> I think in general we should backport only as necessary, and leave
>>> it
>>> > on
>>> > >>> the committers to decide if backporting a particular change is
>>> > necessary.
>>> > >>>
>>> > >>>
>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>> > >>>
>>> > >>>> This is exactly where our views differ, Ben : )
>>> > >>>>
>>> > >>>> Ideally, I would like a release manager to have more ownership and
>>> > less
>>> > >>>> manual work. In my imagination, a release manager has more power
>>> and
>>> > >>>> control about dates, features, backports and everything that is
>>> > related
>>> > >>>> to
>>> > >>>> "their" branch. I would also like us to back port as little as
>>> > >>>> possible, to
>>> > >>>> simplify testing and releasing patch versions.
>>> > >>>>
>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>>> bmahler@apache.org>
>>> > >>>> wrote:
>>> > >>>>
>>> > >>>> +user, I probably it would be good to hear from users as well.
>>> > >>>>>
>>> > >>>>> Please see the original proposal as well as Alex's proposal and
>>> let
>>> > us
>>> > >>>>> know
>>> > >>>>> your thoughts.
>>> > >>>>>
>>> > >>>>> To continue the discussion from where Alex left off:
>>> > >>>>>
>>> > >>>>> > Other bugs and significant improvements, e.g., performance,
>>> may be
>>> > >>>>> back
>>> > >>>>> ported,
>>> > >>>>> the release manager should ideally be the one who decides on
>>> this.
>>> > >>>>>
>>> > >>>>> I'm a little puzzled by this, why is the release manager
>>> involved? As
>>> > >>>>> we
>>> > >>>>> already document, backports occur when the bug is fixed, so this
>>> > >>>>> happens in
>>> > >>>>> the steady state of development, not at release time. The release
>>> > >>>>> manager
>>> > >>>>> only comes in at the time of the release itself, at which point
>>> all
>>> > >>>>> backports have already happened and the release manager handles
>>> the
>>> > >>>>> release
>>> > >>>>> process. Only blocker level issues can stop the release and
>>> while the
>>> > >>>>> release manager has a strong say, we should generally agree on
>>> what
>>> > >>>>> consists of a release blocking issue.
>>> > >>>>>
>>> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
>>> > >>>>> commit
>>> > >>>>> that applies cleanly, right after I commit it to master (with the
>>> > >>>>> exceptions I listed below).
>>> > >>>>>
>>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>>> alex@mesosphere.com
>>> > >
>>> > >>>>> wrote:
>>> > >>>>>
>>> > >>>>> > I would like to back port as little as possible. I suggest the
>>> > >>>>> following
>>> > >>>>> > criteria:
>>> > >>>>> >
>>> > >>>>> > * By default, regressions are back ported to existing release
>>> > >>>>> branches. A
>>> > >>>>> > bug is considered a regression if the functionality is present
>>> in
>>> > the
>>> > >>>>> > previous minor or patch version and is not affected by the bug
>>> > there.
>>> > >>>>> >
>>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>> > >>>>> >
>>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>>> may
>>> > be
>>> > >>>>> back
>>> > >>>>> > ported, the release manager should ideally be the one who
>>> decides
>>> > on
>>> > >>>>> this.
>>> > >>>>> >
>>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>>> vinodkone@apache.org
>>> > >
>>> > >>>>> wrote:
>>> > >>>>> >
>>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>> > >>>>> points you
>>> > >>>>> > > made.
>>> > >>>>> > >
>>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>>> > >>>>> > >
>>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>> > >>>>> bmahler@apache.org>
>>> > >>>>> > > wrote:
>>> > >>>>> > >
>>> > >>>>> > > > I realized recently that we aren't all on the same page
>>> with
>>> > >>>>> > backporting.
>>> > >>>>> > > > We currently only document the following:
>>> > >>>>> > > >
>>> > >>>>> > > > "Typically the fix for an issue that is affecting supported
>>> > >>>>> releases
>>> > >>>>> > > lands
>>> > >>>>> > > > on the master branch and is then backported to the release
>>> > >>>>> branch(es).
>>> > >>>>> > In
>>> > >>>>> > > > rare cases, the fix might directly go into a release branch
>>> > >>>>> without
>>> > >>>>> > > landing
>>> > >>>>> > > > on master (e.g., fix / issue is not applicable to
>>> master)." [1]
>>> > >>>>> > > >
>>> > >>>>> > > > This leaves room for interpretation about what lies
>>> outside of
>>> > >>>>> > "typical".
>>> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
>>> I'd
>>> > >>>>> like
>>> > >>>>> to
>>> > >>>>> > > hear
>>> > >>>>> > > > what others have in mind:
>>> > >>>>> > > >
>>> > >>>>> > > > * By default, bug fixes at any level should be backported
>>> to
>>> > >>>>> existing
>>> > >>>>> > > > release branches if it affects those releases. Especially
>>> > >>>>> important:
>>> > >>>>> > > > crashes, bugs in non-experimental features.
>>> > >>>>> > > >
>>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>> > >>>>> backport
>>> > >>>>> > > fixes
>>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs
>>> in
>>> > >>>>> > experimental
>>> > >>>>> > > > features.
>>> > >>>>> > > >
>>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>>> performance
>>> > >>>>> > > > improvements.
>>> > >>>>> > > >
>>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>>> terms
>>> > of
>>> > >>>>> which
>>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>>> > policy
>>> > >>>>> that
>>> > >>>>> > > gives
>>> > >>>>> > > > everyone the right mindset for common cases and then
>>> discuss
>>> > >>>>> corner
>>> > >>>>> > cases
>>> > >>>>> > > > on-demand in the future.
>>> > >>>>> > > >
>>> > >>>>> > > > [1] http://mesos.apache.org/docume
>>> ntation/latest/versioning/
>>> > >>>>> > > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>>
>>> > >>>>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: Backport Policy
Posted by Benjamin Mahler <bm...@apache.org>.
Thanks to everyone who chimed in. To Gilbert's questions, I think that's
outside the territory of what a policy or guidelines will cover, and will
be up to the judgement of the committer.
My initial search for "backport" was insufficient and I missed the
following existing comment in a different section of versioning.md lays out
a policy of only critical issues:
<start quote>
At any given time, 3 releases are supported: ... Support means fixing of
critical issues that affect the release. Once an issue is deemed critical,
it will be fixed in only those affected releases that are still supported.
...
Which issues are considered critical?
Security fixes
Compatibility regressions
Functional regressions
Performance regressions
Fixes for 3rd party integration (e.g., Docker remote API)
Whether an issue is considered critical or not is sometimes subjective. In
some cases it is obvious and sometimes it is fuzzy. Users should work with
committers to figure out the criticality of an issue and get agreement and
commitment for support.
<end quote>
From the discussion here, it sounds like we should add the guidelines
similar to what Greg suggested above to clarify that there's some
subjectivity here, as well as keeping some of the clear cases (like what
the existing documentation does). I can send out a change sometime soon.
On Wed, Jul 18, 2018 at 10:37 AM, Gilbert Song <gi...@mesosphere.io>
wrote:
> Thanks for clarifying the backporting policy, BenM!
>
> I totally agree with the changes proposed for the backporting policy, but
> I realize two more scenarios that are more clear to me yet:
>
> - There are some bugs that are not fixable (due to legacy technical
> decisions), and we end up with fixing the issue by a semantic/behavior
> change in a new release. Do we expect this semantic/behavior change being
> backported?
> - There might be some bugs that root cause is unknown yet, but it did
> impact on a couple releases. If we decide to add some commits for debugging
> purpose (e.g., a new debugging endpoint, or more logging), should we also
> allow these patches to be backported?
>
> For #2, I think we should do the backporting, but for #1, maybe more
> discussion is needed since it relates to whether the user has to upgrade or
> not.
>
> Cheers,
> Gilbert
>
> On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
>
>> I don’t have a big stake in, however, one opinion is if a large
>> commercial enterprise was using a specific release that is working the
>> desire is often to only upgrade if necessary. Necessary can be for a
>> number of reasons including new features; however if a new feature is not
>> needed the compelling reason to upgrade is to fix a specific problem that
>> is causing issues. Thus keeping a maintenance release stable is very
>> important and reducing the chance for, while fixing one problem,
>> introducing another.
>>
>> Often a clear classification of severity of the problem would dictate the
>> need to make a change. (yes these can be subjective, but some guidance
>> would be better than nothing).
>>
>> It might be good to give committers guidance on back porting things that
>> have a high impact on improving a problem. Fixing a crashing bug, fixing a
>> degenerative performance issue, etc, where these issues have no easy/viable
>> work around. Nice to have fixes aren’t, always, worth updating to.
>>
>> There can be an argument to respond with a “then don’t upgrade” but if
>> changing the release with “nice to have’s” and several point releases later
>> when a critical bug is fixed then the org if forced to accept the risk of
>> the nice to have’s.
>>
>> just an opinion.
>> …larry
>>
>>
>> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
>> wrote:
>>
>> I just have a comment on a special case:
>> If a fix for a flaky test is easy to backport,
>> IMO we probably should backport it,
>> otherwise if someone backports another critical fix in the future,
>> it would take them extra effort to check all CI failures.
>>
>> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
>>
>>> I like how you summarized it Greg and I would vote for leaving the
>>> decision
>>> to the committer too. In addition to what others mentioned, I think
>>> committer should've the responsibility because if things break in a point
>>> release (after it is released), it is the committer and contributor who
>>> are
>>> on the hook to triage and fix it and not the release manager.
>>>
>>> Having said that, if "during" the release process (i.e., cutting an RC)
>>> these backports cause delays for a release manager in getting the release
>>> out (e.g., CI flakiness introduced due to backports), release manager
>>> could
>>> be the ultimate arbiter on whether such a backport should be reverted or
>>> fixed by the committer/contributor. Hopefully such issues are caught much
>>> before a release process is started (e.g., CI running against release
>>> branches).
>>>
>>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>>
>>> > Greg, I like your idea of adding a prescriptive "policy" when
>>> evaluating
>>> > whether a bug fix should be backported, and leave the decision to
>>> committer
>>> > (because they have the most context, and avoid a bottleneck in the
>>> > process).
>>> >
>>> > - Jie
>>> >
>>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io>
>>> wrote:
>>> >
>>> > > My impression is that we have two opposing schools of thought here:
>>> > >
>>> > > 1. Backport as little as possible, to avoid unforeseen
>>> consequences
>>> > > 2. Backport as much as proves practical, to eliminate bugs in
>>> > > supported versions
>>> > >
>>> > > Do other people agree with this assessment?
>>> > >
>>> > > If so, how can we find common ground? One possible solution would be
>>> to
>>> > > leave the decision on backporting up to the committer, without
>>> > specifying a
>>> > > project-wide policy. This seems to be the status quo, and would lead
>>> to
>>> > > some variation across committers regarding what types of fixes are
>>> > > backported. We could also choose to delegate the decision to the
>>> release
>>> > > manager; I favor leaving the decision with the committer, to
>>> eliminate
>>> > the
>>> > > burden on release managers.
>>> > >
>>> > > Here's a thought: rather than defining a prescriptive "policy" that
>>> we
>>> > > expect committers to abide by, we could enumerate in the
>>> documentation
>>> > the
>>> > > competing concerns that we expect committers to consider when making
>>> > > decisions on backports. The committing docs could read something
>>> like:
>>> > >
>>> > > "When bug fixes are committed to master, the committer should
>>> evaluate
>>> > the
>>> > > fix to determine whether or not it should be backported to supported
>>> > > versions. This is left to the committer, but they are expected to
>>> weigh
>>> > the
>>> > > following concerns when making the decision:
>>> > >
>>> > > - Every backported change comes with a risk of unintended
>>> > > consequences. The change should be carefully evaluated to ensure
>>> that
>>> > such
>>> > > side-effects are highly unlikely.
>>> > > - As the complexity of applying a backport increases due to merge
>>> > > conflicts, the likelihood of unintended consequences also
>>> increases.
>>> > Bug
>>> > > fixes which require extensive rebasing should only be backported
>>> when
>>> > the
>>> > > bug is critical enough to warrant the risk.
>>> > > - Users of supported versions benefit greatly from the resolution
>>> of
>>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>>> > allayed
>>> > > for a given bug fix, it should be backported."
>>> > >
>>> > >
>>> > > Cheers,
>>> > > Greg
>>> > >
>>> > >
>>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <alex@mesosphere.com
>>> >
>>> > > wrote:
>>> > >
>>> > >> Back porting as little as possible is the ultimate goal for me. My
>>> > >> reasons are closely aligned with what Andrew wrote above.
>>> > >>
>>> > >> If we agree on this strategy, the next question is how to enforce
>>> it. My
>>> > >> intuition is that committers will lean towards back porting their
>>> > patches
>>> > >> in arguable cases, because humans tend to overestimate the
>>> importance of
>>> > >> their personal work. Delegating the decision in such cases to a
>>> release
>>> > >> manager in my opinion will help us enforce the strategy of minimal
>>> > number
>>> > >> backports. As a bonus, the release manager will have a much better
>>> > >> understanding of what's going on with the release, keyword: "more
>>> > >> ownership".
>>> > >>
>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>>> > >> andrew@schwartzmeyer.com> wrote:
>>> > >>
>>> > >>> I believe I fall somewhere between Alex and Ben.
>>> > >>>
>>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>>> of
>>> > >>> backporting as little as possible (and agree with his criteria). My
>>> > >>> reasoning is that all changes can have unforeseen consequences,
>>> which I
>>> > >>> believe is something to be actively avoided in already released
>>> > versions.
>>> > >>> The reason for backporting patches to fix regressions is the same
>>> as
>>> > the
>>> > >>> reason to avoid backporting as much as possible: keep behavior
>>> > consistent
>>> > >>> (and safe) within a release. With that as the goal of a branch in
>>> > >>> maintenance mode, it makes sense to fix regressions, and make
>>> > exceptions to
>>> > >>> fix CVEs and other critical/blocking issues.
>>> > >>>
>>> > >>> As for who should decide what to backport, I lean toward Ben's
>>> view of
>>> > >>> the burden being on the committer. I don't think we should add more
>>> > work
>>> > >>> for release managers, and I think the committer/shepherd obviously
>>> has
>>> > the
>>> > >>> most understanding of the context around changes proposed for
>>> backport.
>>> > >>>
>>> > >>> Here's an example of a recent bugfix which I backported:
>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>> > >>>
>>> > >>> While normally I believe this change falls under "avoid due to
>>> > >>> unforeseen consequences," I made an exception as the bug was old,
>>> circa
>>> > >>> 2015, (indicating it had been an issue for others), and was causing
>>> > >>> recurring failures in testing. The fix itself was very small,
>>> meaning
>>> > it
>>> > >>> was easier to evaluate for possible side effects, so I felt a
>>> little
>>> > safer
>>> > >>> in that regard. The effect of not having the fix was a fatal and
>>> > undesired
>>> > >>> crash, which furthermore left troublesome side effects on the
>>> system
>>> > (you
>>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>>> > (DC/OS)
>>> > >>> wanted it in their next bump, which necessitated backporting to the
>>> > release
>>> > >>> they were pulling in.
>>> > >>>
>>> > >>> I think in general we should backport only as necessary, and leave
>>> it
>>> > on
>>> > >>> the committers to decide if backporting a particular change is
>>> > necessary.
>>> > >>>
>>> > >>>
>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>> > >>>
>>> > >>>> This is exactly where our views differ, Ben : )
>>> > >>>>
>>> > >>>> Ideally, I would like a release manager to have more ownership and
>>> > less
>>> > >>>> manual work. In my imagination, a release manager has more power
>>> and
>>> > >>>> control about dates, features, backports and everything that is
>>> > related
>>> > >>>> to
>>> > >>>> "their" branch. I would also like us to back port as little as
>>> > >>>> possible, to
>>> > >>>> simplify testing and releasing patch versions.
>>> > >>>>
>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>>> bmahler@apache.org>
>>> > >>>> wrote:
>>> > >>>>
>>> > >>>> +user, I probably it would be good to hear from users as well.
>>> > >>>>>
>>> > >>>>> Please see the original proposal as well as Alex's proposal and
>>> let
>>> > us
>>> > >>>>> know
>>> > >>>>> your thoughts.
>>> > >>>>>
>>> > >>>>> To continue the discussion from where Alex left off:
>>> > >>>>>
>>> > >>>>> > Other bugs and significant improvements, e.g., performance,
>>> may be
>>> > >>>>> back
>>> > >>>>> ported,
>>> > >>>>> the release manager should ideally be the one who decides on
>>> this.
>>> > >>>>>
>>> > >>>>> I'm a little puzzled by this, why is the release manager
>>> involved? As
>>> > >>>>> we
>>> > >>>>> already document, backports occur when the bug is fixed, so this
>>> > >>>>> happens in
>>> > >>>>> the steady state of development, not at release time. The release
>>> > >>>>> manager
>>> > >>>>> only comes in at the time of the release itself, at which point
>>> all
>>> > >>>>> backports have already happened and the release manager handles
>>> the
>>> > >>>>> release
>>> > >>>>> process. Only blocker level issues can stop the release and
>>> while the
>>> > >>>>> release manager has a strong say, we should generally agree on
>>> what
>>> > >>>>> consists of a release blocking issue.
>>> > >>>>>
>>> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
>>> > >>>>> commit
>>> > >>>>> that applies cleanly, right after I commit it to master (with the
>>> > >>>>> exceptions I listed below).
>>> > >>>>>
>>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>>> alex@mesosphere.com
>>> > >
>>> > >>>>> wrote:
>>> > >>>>>
>>> > >>>>> > I would like to back port as little as possible. I suggest the
>>> > >>>>> following
>>> > >>>>> > criteria:
>>> > >>>>> >
>>> > >>>>> > * By default, regressions are back ported to existing release
>>> > >>>>> branches. A
>>> > >>>>> > bug is considered a regression if the functionality is present
>>> in
>>> > the
>>> > >>>>> > previous minor or patch version and is not affected by the bug
>>> > there.
>>> > >>>>> >
>>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>> > >>>>> >
>>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>>> may
>>> > be
>>> > >>>>> back
>>> > >>>>> > ported, the release manager should ideally be the one who
>>> decides
>>> > on
>>> > >>>>> this.
>>> > >>>>> >
>>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>>> vinodkone@apache.org
>>> > >
>>> > >>>>> wrote:
>>> > >>>>> >
>>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>> > >>>>> points you
>>> > >>>>> > > made.
>>> > >>>>> > >
>>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>>> > >>>>> > >
>>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>> > >>>>> bmahler@apache.org>
>>> > >>>>> > > wrote:
>>> > >>>>> > >
>>> > >>>>> > > > I realized recently that we aren't all on the same page
>>> with
>>> > >>>>> > backporting.
>>> > >>>>> > > > We currently only document the following:
>>> > >>>>> > > >
>>> > >>>>> > > > "Typically the fix for an issue that is affecting supported
>>> > >>>>> releases
>>> > >>>>> > > lands
>>> > >>>>> > > > on the master branch and is then backported to the release
>>> > >>>>> branch(es).
>>> > >>>>> > In
>>> > >>>>> > > > rare cases, the fix might directly go into a release branch
>>> > >>>>> without
>>> > >>>>> > > landing
>>> > >>>>> > > > on master (e.g., fix / issue is not applicable to
>>> master)." [1]
>>> > >>>>> > > >
>>> > >>>>> > > > This leaves room for interpretation about what lies
>>> outside of
>>> > >>>>> > "typical".
>>> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
>>> I'd
>>> > >>>>> like
>>> > >>>>> to
>>> > >>>>> > > hear
>>> > >>>>> > > > what others have in mind:
>>> > >>>>> > > >
>>> > >>>>> > > > * By default, bug fixes at any level should be backported
>>> to
>>> > >>>>> existing
>>> > >>>>> > > > release branches if it affects those releases. Especially
>>> > >>>>> important:
>>> > >>>>> > > > crashes, bugs in non-experimental features.
>>> > >>>>> > > >
>>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>> > >>>>> backport
>>> > >>>>> > > fixes
>>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs
>>> in
>>> > >>>>> > experimental
>>> > >>>>> > > > features.
>>> > >>>>> > > >
>>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>>> performance
>>> > >>>>> > > > improvements.
>>> > >>>>> > > >
>>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>>> terms
>>> > of
>>> > >>>>> which
>>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>>> > policy
>>> > >>>>> that
>>> > >>>>> > > gives
>>> > >>>>> > > > everyone the right mindset for common cases and then
>>> discuss
>>> > >>>>> corner
>>> > >>>>> > cases
>>> > >>>>> > > > on-demand in the future.
>>> > >>>>> > > >
>>> > >>>>> > > > [1] http://mesos.apache.org/docume
>>> ntation/latest/versioning/
>>> > >>>>> > > >
>>> > >>>>> > >
>>> > >>>>> >
>>> > >>>>>
>>> > >>>>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: Backport Policy
Posted by Gilbert Song <gi...@mesosphere.io>.
Thanks for clarifying the backporting policy, BenM!
I totally agree with the changes proposed for the backporting policy, but I
realize two more scenarios that are more clear to me yet:
- There are some bugs that are not fixable (due to legacy technical
decisions), and we end up with fixing the issue by a semantic/behavior
change in a new release. Do we expect this semantic/behavior change being
backported?
- There might be some bugs that root cause is unknown yet, but it did
impact on a couple releases. If we decide to add some commits for debugging
purpose (e.g., a new debugging endpoint, or more logging), should we also
allow these patches to be backported?
For #2, I think we should do the backporting, but for #1, maybe more
discussion is needed since it relates to whether the user has to upgrade or
not.
Cheers,
Gilbert
On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
> I don’t have a big stake in, however, one opinion is if a large commercial
> enterprise was using a specific release that is working the desire is often
> to only upgrade if necessary. Necessary can be for a number of reasons
> including new features; however if a new feature is not needed the
> compelling reason to upgrade is to fix a specific problem that is causing
> issues. Thus keeping a maintenance release stable is very important and
> reducing the chance for, while fixing one problem, introducing another.
>
> Often a clear classification of severity of the problem would dictate the
> need to make a change. (yes these can be subjective, but some guidance
> would be better than nothing).
>
> It might be good to give committers guidance on back porting things that
> have a high impact on improving a problem. Fixing a crashing bug, fixing a
> degenerative performance issue, etc, where these issues have no easy/viable
> work around. Nice to have fixes aren’t, always, worth updating to.
>
> There can be an argument to respond with a “then don’t upgrade” but if
> changing the release with “nice to have’s” and several point releases later
> when a critical bug is fixed then the org if forced to accept the risk of
> the nice to have’s.
>
> just an opinion.
> …larry
>
>
> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
> wrote:
>
> I just have a comment on a special case:
> If a fix for a flaky test is easy to backport,
> IMO we probably should backport it,
> otherwise if someone backports another critical fix in the future,
> it would take them extra effort to check all CI failures.
>
> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
>
>> I like how you summarized it Greg and I would vote for leaving the
>> decision
>> to the committer too. In addition to what others mentioned, I think
>> committer should've the responsibility because if things break in a point
>> release (after it is released), it is the committer and contributor who
>> are
>> on the hook to triage and fix it and not the release manager.
>>
>> Having said that, if "during" the release process (i.e., cutting an RC)
>> these backports cause delays for a release manager in getting the release
>> out (e.g., CI flakiness introduced due to backports), release manager
>> could
>> be the ultimate arbiter on whether such a backport should be reverted or
>> fixed by the committer/contributor. Hopefully such issues are caught much
>> before a release process is started (e.g., CI running against release
>> branches).
>>
>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>
>> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
>> > whether a bug fix should be backported, and leave the decision to
>> committer
>> > (because they have the most context, and avoid a bottleneck in the
>> > process).
>> >
>> > - Jie
>> >
>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
>> >
>> > > My impression is that we have two opposing schools of thought here:
>> > >
>> > > 1. Backport as little as possible, to avoid unforeseen consequences
>> > > 2. Backport as much as proves practical, to eliminate bugs in
>> > > supported versions
>> > >
>> > > Do other people agree with this assessment?
>> > >
>> > > If so, how can we find common ground? One possible solution would be
>> to
>> > > leave the decision on backporting up to the committer, without
>> > specifying a
>> > > project-wide policy. This seems to be the status quo, and would lead
>> to
>> > > some variation across committers regarding what types of fixes are
>> > > backported. We could also choose to delegate the decision to the
>> release
>> > > manager; I favor leaving the decision with the committer, to eliminate
>> > the
>> > > burden on release managers.
>> > >
>> > > Here's a thought: rather than defining a prescriptive "policy" that we
>> > > expect committers to abide by, we could enumerate in the documentation
>> > the
>> > > competing concerns that we expect committers to consider when making
>> > > decisions on backports. The committing docs could read something like:
>> > >
>> > > "When bug fixes are committed to master, the committer should evaluate
>> > the
>> > > fix to determine whether or not it should be backported to supported
>> > > versions. This is left to the committer, but they are expected to
>> weigh
>> > the
>> > > following concerns when making the decision:
>> > >
>> > > - Every backported change comes with a risk of unintended
>> > > consequences. The change should be carefully evaluated to ensure
>> that
>> > such
>> > > side-effects are highly unlikely.
>> > > - As the complexity of applying a backport increases due to merge
>> > > conflicts, the likelihood of unintended consequences also
>> increases.
>> > Bug
>> > > fixes which require extensive rebasing should only be backported
>> when
>> > the
>> > > bug is critical enough to warrant the risk.
>> > > - Users of supported versions benefit greatly from the resolution
>> of
>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>> > allayed
>> > > for a given bug fix, it should be backported."
>> > >
>> > >
>> > > Cheers,
>> > > Greg
>> > >
>> > >
>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
>> > > wrote:
>> > >
>> > >> Back porting as little as possible is the ultimate goal for me. My
>> > >> reasons are closely aligned with what Andrew wrote above.
>> > >>
>> > >> If we agree on this strategy, the next question is how to enforce
>> it. My
>> > >> intuition is that committers will lean towards back porting their
>> > patches
>> > >> in arguable cases, because humans tend to overestimate the
>> importance of
>> > >> their personal work. Delegating the decision in such cases to a
>> release
>> > >> manager in my opinion will help us enforce the strategy of minimal
>> > number
>> > >> backports. As a bonus, the release manager will have a much better
>> > >> understanding of what's going on with the release, keyword: "more
>> > >> ownership".
>> > >>
>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> > >> andrew@schwartzmeyer.com> wrote:
>> > >>
>> > >>> I believe I fall somewhere between Alex and Ben.
>> > >>>
>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>> of
>> > >>> backporting as little as possible (and agree with his criteria). My
>> > >>> reasoning is that all changes can have unforeseen consequences,
>> which I
>> > >>> believe is something to be actively avoided in already released
>> > versions.
>> > >>> The reason for backporting patches to fix regressions is the same as
>> > the
>> > >>> reason to avoid backporting as much as possible: keep behavior
>> > consistent
>> > >>> (and safe) within a release. With that as the goal of a branch in
>> > >>> maintenance mode, it makes sense to fix regressions, and make
>> > exceptions to
>> > >>> fix CVEs and other critical/blocking issues.
>> > >>>
>> > >>> As for who should decide what to backport, I lean toward Ben's view
>> of
>> > >>> the burden being on the committer. I don't think we should add more
>> > work
>> > >>> for release managers, and I think the committer/shepherd obviously
>> has
>> > the
>> > >>> most understanding of the context around changes proposed for
>> backport.
>> > >>>
>> > >>> Here's an example of a recent bugfix which I backported:
>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>> > >>>
>> > >>> While normally I believe this change falls under "avoid due to
>> > >>> unforeseen consequences," I made an exception as the bug was old,
>> circa
>> > >>> 2015, (indicating it had been an issue for others), and was causing
>> > >>> recurring failures in testing. The fix itself was very small,
>> meaning
>> > it
>> > >>> was easier to evaluate for possible side effects, so I felt a little
>> > safer
>> > >>> in that regard. The effect of not having the fix was a fatal and
>> > undesired
>> > >>> crash, which furthermore left troublesome side effects on the system
>> > (you
>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>> > (DC/OS)
>> > >>> wanted it in their next bump, which necessitated backporting to the
>> > release
>> > >>> they were pulling in.
>> > >>>
>> > >>> I think in general we should backport only as necessary, and leave
>> it
>> > on
>> > >>> the committers to decide if backporting a particular change is
>> > necessary.
>> > >>>
>> > >>>
>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>> > >>>
>> > >>>> This is exactly where our views differ, Ben : )
>> > >>>>
>> > >>>> Ideally, I would like a release manager to have more ownership and
>> > less
>> > >>>> manual work. In my imagination, a release manager has more power
>> and
>> > >>>> control about dates, features, backports and everything that is
>> > related
>> > >>>> to
>> > >>>> "their" branch. I would also like us to back port as little as
>> > >>>> possible, to
>> > >>>> simplify testing and releasing patch versions.
>> > >>>>
>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>> bmahler@apache.org>
>> > >>>> wrote:
>> > >>>>
>> > >>>> +user, I probably it would be good to hear from users as well.
>> > >>>>>
>> > >>>>> Please see the original proposal as well as Alex's proposal and
>> let
>> > us
>> > >>>>> know
>> > >>>>> your thoughts.
>> > >>>>>
>> > >>>>> To continue the discussion from where Alex left off:
>> > >>>>>
>> > >>>>> > Other bugs and significant improvements, e.g., performance, may
>> be
>> > >>>>> back
>> > >>>>> ported,
>> > >>>>> the release manager should ideally be the one who decides on this.
>> > >>>>>
>> > >>>>> I'm a little puzzled by this, why is the release manager
>> involved? As
>> > >>>>> we
>> > >>>>> already document, backports occur when the bug is fixed, so this
>> > >>>>> happens in
>> > >>>>> the steady state of development, not at release time. The release
>> > >>>>> manager
>> > >>>>> only comes in at the time of the release itself, at which point
>> all
>> > >>>>> backports have already happened and the release manager handles
>> the
>> > >>>>> release
>> > >>>>> process. Only blocker level issues can stop the release and while
>> the
>> > >>>>> release manager has a strong say, we should generally agree on
>> what
>> > >>>>> consists of a release blocking issue.
>> > >>>>>
>> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
>> > >>>>> commit
>> > >>>>> that applies cleanly, right after I commit it to master (with the
>> > >>>>> exceptions I listed below).
>> > >>>>>
>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>> alex@mesosphere.com
>> > >
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>> > I would like to back port as little as possible. I suggest the
>> > >>>>> following
>> > >>>>> > criteria:
>> > >>>>> >
>> > >>>>> > * By default, regressions are back ported to existing release
>> > >>>>> branches. A
>> > >>>>> > bug is considered a regression if the functionality is present
>> in
>> > the
>> > >>>>> > previous minor or patch version and is not affected by the bug
>> > there.
>> > >>>>> >
>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> > >>>>> >
>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>> may
>> > be
>> > >>>>> back
>> > >>>>> > ported, the release manager should ideally be the one who
>> decides
>> > on
>> > >>>>> this.
>> > >>>>> >
>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>> vinodkone@apache.org
>> > >
>> > >>>>> wrote:
>> > >>>>> >
>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>> > >>>>> points you
>> > >>>>> > > made.
>> > >>>>> > >
>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>> > >>>>> > >
>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>> > >>>>> bmahler@apache.org>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > > > I realized recently that we aren't all on the same page with
>> > >>>>> > backporting.
>> > >>>>> > > > We currently only document the following:
>> > >>>>> > > >
>> > >>>>> > > > "Typically the fix for an issue that is affecting supported
>> > >>>>> releases
>> > >>>>> > > lands
>> > >>>>> > > > on the master branch and is then backported to the release
>> > >>>>> branch(es).
>> > >>>>> > In
>> > >>>>> > > > rare cases, the fix might directly go into a release branch
>> > >>>>> without
>> > >>>>> > > landing
>> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)."
>> [1]
>> > >>>>> > > >
>> > >>>>> > > > This leaves room for interpretation about what lies outside
>> of
>> > >>>>> > "typical".
>> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
>> I'd
>> > >>>>> like
>> > >>>>> to
>> > >>>>> > > hear
>> > >>>>> > > > what others have in mind:
>> > >>>>> > > >
>> > >>>>> > > > * By default, bug fixes at any level should be backported to
>> > >>>>> existing
>> > >>>>> > > > release branches if it affects those releases. Especially
>> > >>>>> important:
>> > >>>>> > > > crashes, bugs in non-experimental features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
>> > >>>>> backport
>> > >>>>> > > fixes
>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>> > >>>>> > experimental
>> > >>>>> > > > features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>> performance
>> > >>>>> > > > improvements.
>> > >>>>> > > >
>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>> terms
>> > of
>> > >>>>> which
>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>> > policy
>> > >>>>> that
>> > >>>>> > > gives
>> > >>>>> > > > everyone the right mindset for common cases and then discuss
>> > >>>>> corner
>> > >>>>> > cases
>> > >>>>> > > > on-demand in the future.
>> > >>>>> > > >
>> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/
>> versioning/
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> > >>>>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: Backport Policy
Posted by Gilbert Song <gi...@mesosphere.io>.
Thanks for clarifying the backporting policy, BenM!
I totally agree with the changes proposed for the backporting policy, but I
realize two more scenarios that are more clear to me yet:
- There are some bugs that are not fixable (due to legacy technical
decisions), and we end up with fixing the issue by a semantic/behavior
change in a new release. Do we expect this semantic/behavior change being
backported?
- There might be some bugs that root cause is unknown yet, but it did
impact on a couple releases. If we decide to add some commits for debugging
purpose (e.g., a new debugging endpoint, or more logging), should we also
allow these patches to be backported?
For #2, I think we should do the backporting, but for #1, maybe more
discussion is needed since it relates to whether the user has to upgrade or
not.
Cheers,
Gilbert
On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <la...@mac.com> wrote:
> I don’t have a big stake in, however, one opinion is if a large commercial
> enterprise was using a specific release that is working the desire is often
> to only upgrade if necessary. Necessary can be for a number of reasons
> including new features; however if a new feature is not needed the
> compelling reason to upgrade is to fix a specific problem that is causing
> issues. Thus keeping a maintenance release stable is very important and
> reducing the chance for, while fixing one problem, introducing another.
>
> Often a clear classification of severity of the problem would dictate the
> need to make a change. (yes these can be subjective, but some guidance
> would be better than nothing).
>
> It might be good to give committers guidance on back porting things that
> have a high impact on improving a problem. Fixing a crashing bug, fixing a
> degenerative performance issue, etc, where these issues have no easy/viable
> work around. Nice to have fixes aren’t, always, worth updating to.
>
> There can be an argument to respond with a “then don’t upgrade” but if
> changing the release with “nice to have’s” and several point releases later
> when a critical bug is fixed then the org if forced to accept the risk of
> the nice to have’s.
>
> just an opinion.
> …larry
>
>
> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io>
> wrote:
>
> I just have a comment on a special case:
> If a fix for a flaky test is easy to backport,
> IMO we probably should backport it,
> otherwise if someone backports another critical fix in the future,
> it would take them extra effort to check all CI failures.
>
> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
>
>> I like how you summarized it Greg and I would vote for leaving the
>> decision
>> to the committer too. In addition to what others mentioned, I think
>> committer should've the responsibility because if things break in a point
>> release (after it is released), it is the committer and contributor who
>> are
>> on the hook to triage and fix it and not the release manager.
>>
>> Having said that, if "during" the release process (i.e., cutting an RC)
>> these backports cause delays for a release manager in getting the release
>> out (e.g., CI flakiness introduced due to backports), release manager
>> could
>> be the ultimate arbiter on whether such a backport should be reverted or
>> fixed by the committer/contributor. Hopefully such issues are caught much
>> before a release process is started (e.g., CI running against release
>> branches).
>>
>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>>
>> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
>> > whether a bug fix should be backported, and leave the decision to
>> committer
>> > (because they have the most context, and avoid a bottleneck in the
>> > process).
>> >
>> > - Jie
>> >
>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
>> >
>> > > My impression is that we have two opposing schools of thought here:
>> > >
>> > > 1. Backport as little as possible, to avoid unforeseen consequences
>> > > 2. Backport as much as proves practical, to eliminate bugs in
>> > > supported versions
>> > >
>> > > Do other people agree with this assessment?
>> > >
>> > > If so, how can we find common ground? One possible solution would be
>> to
>> > > leave the decision on backporting up to the committer, without
>> > specifying a
>> > > project-wide policy. This seems to be the status quo, and would lead
>> to
>> > > some variation across committers regarding what types of fixes are
>> > > backported. We could also choose to delegate the decision to the
>> release
>> > > manager; I favor leaving the decision with the committer, to eliminate
>> > the
>> > > burden on release managers.
>> > >
>> > > Here's a thought: rather than defining a prescriptive "policy" that we
>> > > expect committers to abide by, we could enumerate in the documentation
>> > the
>> > > competing concerns that we expect committers to consider when making
>> > > decisions on backports. The committing docs could read something like:
>> > >
>> > > "When bug fixes are committed to master, the committer should evaluate
>> > the
>> > > fix to determine whether or not it should be backported to supported
>> > > versions. This is left to the committer, but they are expected to
>> weigh
>> > the
>> > > following concerns when making the decision:
>> > >
>> > > - Every backported change comes with a risk of unintended
>> > > consequences. The change should be carefully evaluated to ensure
>> that
>> > such
>> > > side-effects are highly unlikely.
>> > > - As the complexity of applying a backport increases due to merge
>> > > conflicts, the likelihood of unintended consequences also
>> increases.
>> > Bug
>> > > fixes which require extensive rebasing should only be backported
>> when
>> > the
>> > > bug is critical enough to warrant the risk.
>> > > - Users of supported versions benefit greatly from the resolution
>> of
>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
>> > allayed
>> > > for a given bug fix, it should be backported."
>> > >
>> > >
>> > > Cheers,
>> > > Greg
>> > >
>> > >
>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
>> > > wrote:
>> > >
>> > >> Back porting as little as possible is the ultimate goal for me. My
>> > >> reasons are closely aligned with what Andrew wrote above.
>> > >>
>> > >> If we agree on this strategy, the next question is how to enforce
>> it. My
>> > >> intuition is that committers will lean towards back porting their
>> > patches
>> > >> in arguable cases, because humans tend to overestimate the
>> importance of
>> > >> their personal work. Delegating the decision in such cases to a
>> release
>> > >> manager in my opinion will help us enforce the strategy of minimal
>> > number
>> > >> backports. As a bonus, the release manager will have a much better
>> > >> understanding of what's going on with the release, keyword: "more
>> > >> ownership".
>> > >>
>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> > >> andrew@schwartzmeyer.com> wrote:
>> > >>
>> > >>> I believe I fall somewhere between Alex and Ben.
>> > >>>
>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>> of
>> > >>> backporting as little as possible (and agree with his criteria). My
>> > >>> reasoning is that all changes can have unforeseen consequences,
>> which I
>> > >>> believe is something to be actively avoided in already released
>> > versions.
>> > >>> The reason for backporting patches to fix regressions is the same as
>> > the
>> > >>> reason to avoid backporting as much as possible: keep behavior
>> > consistent
>> > >>> (and safe) within a release. With that as the goal of a branch in
>> > >>> maintenance mode, it makes sense to fix regressions, and make
>> > exceptions to
>> > >>> fix CVEs and other critical/blocking issues.
>> > >>>
>> > >>> As for who should decide what to backport, I lean toward Ben's view
>> of
>> > >>> the burden being on the committer. I don't think we should add more
>> > work
>> > >>> for release managers, and I think the committer/shepherd obviously
>> has
>> > the
>> > >>> most understanding of the context around changes proposed for
>> backport.
>> > >>>
>> > >>> Here's an example of a recent bugfix which I backported:
>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>> > >>>
>> > >>> While normally I believe this change falls under "avoid due to
>> > >>> unforeseen consequences," I made an exception as the bug was old,
>> circa
>> > >>> 2015, (indicating it had been an issue for others), and was causing
>> > >>> recurring failures in testing. The fix itself was very small,
>> meaning
>> > it
>> > >>> was easier to evaluate for possible side effects, so I felt a little
>> > safer
>> > >>> in that regard. The effect of not having the fix was a fatal and
>> > undesired
>> > >>> crash, which furthermore left troublesome side effects on the system
>> > (you
>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>> > (DC/OS)
>> > >>> wanted it in their next bump, which necessitated backporting to the
>> > release
>> > >>> they were pulling in.
>> > >>>
>> > >>> I think in general we should backport only as necessary, and leave
>> it
>> > on
>> > >>> the committers to decide if backporting a particular change is
>> > necessary.
>> > >>>
>> > >>>
>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>> > >>>
>> > >>>> This is exactly where our views differ, Ben : )
>> > >>>>
>> > >>>> Ideally, I would like a release manager to have more ownership and
>> > less
>> > >>>> manual work. In my imagination, a release manager has more power
>> and
>> > >>>> control about dates, features, backports and everything that is
>> > related
>> > >>>> to
>> > >>>> "their" branch. I would also like us to back port as little as
>> > >>>> possible, to
>> > >>>> simplify testing and releasing patch versions.
>> > >>>>
>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>> bmahler@apache.org>
>> > >>>> wrote:
>> > >>>>
>> > >>>> +user, I probably it would be good to hear from users as well.
>> > >>>>>
>> > >>>>> Please see the original proposal as well as Alex's proposal and
>> let
>> > us
>> > >>>>> know
>> > >>>>> your thoughts.
>> > >>>>>
>> > >>>>> To continue the discussion from where Alex left off:
>> > >>>>>
>> > >>>>> > Other bugs and significant improvements, e.g., performance, may
>> be
>> > >>>>> back
>> > >>>>> ported,
>> > >>>>> the release manager should ideally be the one who decides on this.
>> > >>>>>
>> > >>>>> I'm a little puzzled by this, why is the release manager
>> involved? As
>> > >>>>> we
>> > >>>>> already document, backports occur when the bug is fixed, so this
>> > >>>>> happens in
>> > >>>>> the steady state of development, not at release time. The release
>> > >>>>> manager
>> > >>>>> only comes in at the time of the release itself, at which point
>> all
>> > >>>>> backports have already happened and the release manager handles
>> the
>> > >>>>> release
>> > >>>>> process. Only blocker level issues can stop the release and while
>> the
>> > >>>>> release manager has a strong say, we should generally agree on
>> what
>> > >>>>> consists of a release blocking issue.
>> > >>>>>
>> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
>> > >>>>> commit
>> > >>>>> that applies cleanly, right after I commit it to master (with the
>> > >>>>> exceptions I listed below).
>> > >>>>>
>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>> alex@mesosphere.com
>> > >
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>> > I would like to back port as little as possible. I suggest the
>> > >>>>> following
>> > >>>>> > criteria:
>> > >>>>> >
>> > >>>>> > * By default, regressions are back ported to existing release
>> > >>>>> branches. A
>> > >>>>> > bug is considered a regression if the functionality is present
>> in
>> > the
>> > >>>>> > previous minor or patch version and is not affected by the bug
>> > there.
>> > >>>>> >
>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> > >>>>> >
>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>> may
>> > be
>> > >>>>> back
>> > >>>>> > ported, the release manager should ideally be the one who
>> decides
>> > on
>> > >>>>> this.
>> > >>>>> >
>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>> vinodkone@apache.org
>> > >
>> > >>>>> wrote:
>> > >>>>> >
>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>> > >>>>> points you
>> > >>>>> > > made.
>> > >>>>> > >
>> > >>>>> > > Once we have consensus, would you mind updating the doc?
>> > >>>>> > >
>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>> > >>>>> bmahler@apache.org>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > > > I realized recently that we aren't all on the same page with
>> > >>>>> > backporting.
>> > >>>>> > > > We currently only document the following:
>> > >>>>> > > >
>> > >>>>> > > > "Typically the fix for an issue that is affecting supported
>> > >>>>> releases
>> > >>>>> > > lands
>> > >>>>> > > > on the master branch and is then backported to the release
>> > >>>>> branch(es).
>> > >>>>> > In
>> > >>>>> > > > rare cases, the fix might directly go into a release branch
>> > >>>>> without
>> > >>>>> > > landing
>> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)."
>> [1]
>> > >>>>> > > >
>> > >>>>> > > > This leaves room for interpretation about what lies outside
>> of
>> > >>>>> > "typical".
>> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
>> I'd
>> > >>>>> like
>> > >>>>> to
>> > >>>>> > > hear
>> > >>>>> > > > what others have in mind:
>> > >>>>> > > >
>> > >>>>> > > > * By default, bug fixes at any level should be backported to
>> > >>>>> existing
>> > >>>>> > > > release branches if it affects those releases. Especially
>> > >>>>> important:
>> > >>>>> > > > crashes, bugs in non-experimental features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
>> > >>>>> backport
>> > >>>>> > > fixes
>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>> > >>>>> > experimental
>> > >>>>> > > > features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional non-bug cases that can be backported:
>> performance
>> > >>>>> > > > improvements.
>> > >>>>> > > >
>> > >>>>> > > > I realize that there is a ton of subtlety here (even in
>> terms
>> > of
>> > >>>>> which
>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
>> > policy
>> > >>>>> that
>> > >>>>> > > gives
>> > >>>>> > > > everyone the right mindset for common cases and then discuss
>> > >>>>> corner
>> > >>>>> > cases
>> > >>>>> > > > on-demand in the future.
>> > >>>>> > > >
>> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/
>> versioning/
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> > >>>>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: Backport Policy
Posted by Lawrence Rau <la...@mac.com>.
I don’t have a big stake in, however, one opinion is if a large commercial enterprise was using a specific release that is working the desire is often to only upgrade if necessary. Necessary can be for a number of reasons including new features; however if a new feature is not needed the compelling reason to upgrade is to fix a specific problem that is causing issues. Thus keeping a maintenance release stable is very important and reducing the chance for, while fixing one problem, introducing another.
Often a clear classification of severity of the problem would dictate the need to make a change. (yes these can be subjective, but some guidance would be better than nothing).
It might be good to give committers guidance on back porting things that have a high impact on improving a problem. Fixing a crashing bug, fixing a degenerative performance issue, etc, where these issues have no easy/viable work around. Nice to have fixes aren’t, always, worth updating to.
There can be an argument to respond with a “then don’t upgrade” but if changing the release with “nice to have’s” and several point releases later when a critical bug is fixed then the org if forced to accept the risk of the nice to have’s.
just an opinion.
…larry
> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <ch...@mesosphere.io> wrote:
>
> I just have a comment on a special case:
> If a fix for a flaky test is easy to backport,
> IMO we probably should backport it,
> otherwise if someone backports another critical fix in the future,
> it would take them extra effort to check all CI failures.
>
> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vinodkone@apache.org <ma...@apache.org>> wrote:
> I like how you summarized it Greg and I would vote for leaving the decision
> to the committer too. In addition to what others mentioned, I think
> committer should've the responsibility because if things break in a point
> release (after it is released), it is the committer and contributor who are
> on the hook to triage and fix it and not the release manager.
>
> Having said that, if "during" the release process (i.e., cutting an RC)
> these backports cause delays for a release manager in getting the release
> out (e.g., CI flakiness introduced due to backports), release manager could
> be the ultimate arbiter on whether such a backport should be reverted or
> fixed by the committer/contributor. Hopefully such issues are caught much
> before a release process is started (e.g., CI running against release
> branches).
>
> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yujie.jay@gmail.com <ma...@gmail.com>> wrote:
>
> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
> > whether a bug fix should be backported, and leave the decision to committer
> > (because they have the most context, and avoid a bottleneck in the
> > process).
> >
> > - Jie
> >
> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <greg@mesosphere.io <ma...@mesosphere.io>> wrote:
> >
> > > My impression is that we have two opposing schools of thought here:
> > >
> > > 1. Backport as little as possible, to avoid unforeseen consequences
> > > 2. Backport as much as proves practical, to eliminate bugs in
> > > supported versions
> > >
> > > Do other people agree with this assessment?
> > >
> > > If so, how can we find common ground? One possible solution would be to
> > > leave the decision on backporting up to the committer, without
> > specifying a
> > > project-wide policy. This seems to be the status quo, and would lead to
> > > some variation across committers regarding what types of fixes are
> > > backported. We could also choose to delegate the decision to the release
> > > manager; I favor leaving the decision with the committer, to eliminate
> > the
> > > burden on release managers.
> > >
> > > Here's a thought: rather than defining a prescriptive "policy" that we
> > > expect committers to abide by, we could enumerate in the documentation
> > the
> > > competing concerns that we expect committers to consider when making
> > > decisions on backports. The committing docs could read something like:
> > >
> > > "When bug fixes are committed to master, the committer should evaluate
> > the
> > > fix to determine whether or not it should be backported to supported
> > > versions. This is left to the committer, but they are expected to weigh
> > the
> > > following concerns when making the decision:
> > >
> > > - Every backported change comes with a risk of unintended
> > > consequences. The change should be carefully evaluated to ensure that
> > such
> > > side-effects are highly unlikely.
> > > - As the complexity of applying a backport increases due to merge
> > > conflicts, the likelihood of unintended consequences also increases.
> > Bug
> > > fixes which require extensive rebasing should only be backported when
> > the
> > > bug is critical enough to warrant the risk.
> > > - Users of supported versions benefit greatly from the resolution of
> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
> > allayed
> > > for a given bug fix, it should be backported."
> > >
> > >
> > > Cheers,
> > > Greg
> > >
> > >
> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <alex@mesosphere.com <ma...@mesosphere.com>>
> > > wrote:
> > >
> > >> Back porting as little as possible is the ultimate goal for me. My
> > >> reasons are closely aligned with what Andrew wrote above.
> > >>
> > >> If we agree on this strategy, the next question is how to enforce it. My
> > >> intuition is that committers will lean towards back porting their
> > patches
> > >> in arguable cases, because humans tend to overestimate the importance of
> > >> their personal work. Delegating the decision in such cases to a release
> > >> manager in my opinion will help us enforce the strategy of minimal
> > number
> > >> backports. As a bonus, the release manager will have a much better
> > >> understanding of what's going on with the release, keyword: "more
> > >> ownership".
> > >>
> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> > >> andrew@schwartzmeyer.com <ma...@schwartzmeyer.com>> wrote:
> > >>
> > >>> I believe I fall somewhere between Alex and Ben.
> > >>>
> > >>> As for deciding what to backport or not, I lean toward Alex's view of
> > >>> backporting as little as possible (and agree with his criteria). My
> > >>> reasoning is that all changes can have unforeseen consequences, which I
> > >>> believe is something to be actively avoided in already released
> > versions.
> > >>> The reason for backporting patches to fix regressions is the same as
> > the
> > >>> reason to avoid backporting as much as possible: keep behavior
> > consistent
> > >>> (and safe) within a release. With that as the goal of a branch in
> > >>> maintenance mode, it makes sense to fix regressions, and make
> > exceptions to
> > >>> fix CVEs and other critical/blocking issues.
> > >>>
> > >>> As for who should decide what to backport, I lean toward Ben's view of
> > >>> the burden being on the committer. I don't think we should add more
> > work
> > >>> for release managers, and I think the committer/shepherd obviously has
> > the
> > >>> most understanding of the context around changes proposed for backport.
> > >>>
> > >>> Here's an example of a recent bugfix which I backported:
> > >>> https://reviews.apache.org/r/67587/ <https://reviews.apache.org/r/67587/> (for MESOS-3790)
> > >>>
> > >>> While normally I believe this change falls under "avoid due to
> > >>> unforeseen consequences," I made an exception as the bug was old, circa
> > >>> 2015, (indicating it had been an issue for others), and was causing
> > >>> recurring failures in testing. The fix itself was very small, meaning
> > it
> > >>> was easier to evaluate for possible side effects, so I felt a little
> > safer
> > >>> in that regard. The effect of not having the fix was a fatal and
> > undesired
> > >>> crash, which furthermore left troublesome side effects on the system
> > (you
> > >>> couldn't bring the agent back up). And lastly, a dependent project
> > (DC/OS)
> > >>> wanted it in their next bump, which necessitated backporting to the
> > release
> > >>> they were pulling in.
> > >>>
> > >>> I think in general we should backport only as necessary, and leave it
> > on
> > >>> the committers to decide if backporting a particular change is
> > necessary.
> > >>>
> > >>>
> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> > >>>
> > >>>> This is exactly where our views differ, Ben : )
> > >>>>
> > >>>> Ideally, I would like a release manager to have more ownership and
> > less
> > >>>> manual work. In my imagination, a release manager has more power and
> > >>>> control about dates, features, backports and everything that is
> > related
> > >>>> to
> > >>>> "their" branch. I would also like us to back port as little as
> > >>>> possible, to
> > >>>> simplify testing and releasing patch versions.
> > >>>>
> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bmahler@apache.org <ma...@apache.org>>
> > >>>> wrote:
> > >>>>
> > >>>> +user, I probably it would be good to hear from users as well.
> > >>>>>
> > >>>>> Please see the original proposal as well as Alex's proposal and let
> > us
> > >>>>> know
> > >>>>> your thoughts.
> > >>>>>
> > >>>>> To continue the discussion from where Alex left off:
> > >>>>>
> > >>>>> > Other bugs and significant improvements, e.g., performance, may be
> > >>>>> back
> > >>>>> ported,
> > >>>>> the release manager should ideally be the one who decides on this.
> > >>>>>
> > >>>>> I'm a little puzzled by this, why is the release manager involved? As
> > >>>>> we
> > >>>>> already document, backports occur when the bug is fixed, so this
> > >>>>> happens in
> > >>>>> the steady state of development, not at release time. The release
> > >>>>> manager
> > >>>>> only comes in at the time of the release itself, at which point all
> > >>>>> backports have already happened and the release manager handles the
> > >>>>> release
> > >>>>> process. Only blocker level issues can stop the release and while the
> > >>>>> release manager has a strong say, we should generally agree on what
> > >>>>> consists of a release blocking issue.
> > >>>>>
> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
> > >>>>> commit
> > >>>>> that applies cleanly, right after I commit it to master (with the
> > >>>>> exceptions I listed below).
> > >>>>>
> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <alex@mesosphere.com <ma...@mesosphere.com>
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>> > I would like to back port as little as possible. I suggest the
> > >>>>> following
> > >>>>> > criteria:
> > >>>>> >
> > >>>>> > * By default, regressions are back ported to existing release
> > >>>>> branches. A
> > >>>>> > bug is considered a regression if the functionality is present in
> > the
> > >>>>> > previous minor or patch version and is not affected by the bug
> > there.
> > >>>>> >
> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >>>>> >
> > >>>>> > * Other bugs and significant improvements, e.g., performance, may
> > be
> > >>>>> back
> > >>>>> > ported, the release manager should ideally be the one who decides
> > on
> > >>>>> this.
> > >>>>> >
> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vinodkone@apache.org <ma...@apache.org>
> > >
> > >>>>> wrote:
> > >>>>> >
> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
> > >>>>> points you
> > >>>>> > > made.
> > >>>>> > >
> > >>>>> > > Once we have consensus, would you mind updating the doc?
> > >>>>> > >
> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
> > >>>>> bmahler@apache.org <ma...@apache.org>>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > > > I realized recently that we aren't all on the same page with
> > >>>>> > backporting.
> > >>>>> > > > We currently only document the following:
> > >>>>> > > >
> > >>>>> > > > "Typically the fix for an issue that is affecting supported
> > >>>>> releases
> > >>>>> > > lands
> > >>>>> > > > on the master branch and is then backported to the release
> > >>>>> branch(es).
> > >>>>> > In
> > >>>>> > > > rare cases, the fix might directly go into a release branch
> > >>>>> without
> > >>>>> > > landing
> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > >>>>> > > >
> > >>>>> > > > This leaves room for interpretation about what lies outside of
> > >>>>> > "typical".
> > >>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
> > >>>>> like
> > >>>>> to
> > >>>>> > > hear
> > >>>>> > > > what others have in mind:
> > >>>>> > > >
> > >>>>> > > > * By default, bug fixes at any level should be backported to
> > >>>>> existing
> > >>>>> > > > release branches if it affects those releases. Especially
> > >>>>> important:
> > >>>>> > > > crashes, bugs in non-experimental features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
> > >>>>> backport
> > >>>>> > > fixes
> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
> > >>>>> > experimental
> > >>>>> > > > features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional non-bug cases that can be backported: performance
> > >>>>> > > > improvements.
> > >>>>> > > >
> > >>>>> > > > I realize that there is a ton of subtlety here (even in terms
> > of
> > >>>>> which
> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
> > policy
> > >>>>> that
> > >>>>> > > gives
> > >>>>> > > > everyone the right mindset for common cases and then discuss
> > >>>>> corner
> > >>>>> > cases
> > >>>>> > > > on-demand in the future.
> > >>>>> > > >
> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/ <http://mesos.apache.org/documentation/latest/versioning/>
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>
> > >>
> > >
> >
Re: Backport Policy
Posted by Chun-Hung Hsiao <ch...@mesosphere.io>.
I just have a comment on a special case:
If a fix for a flaky test is easy to backport,
IMO we probably should backport it,
otherwise if someone backports another critical fix in the future,
it would take them extra effort to check all CI failures.
On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
> I like how you summarized it Greg and I would vote for leaving the decision
> to the committer too. In addition to what others mentioned, I think
> committer should've the responsibility because if things break in a point
> release (after it is released), it is the committer and contributor who are
> on the hook to triage and fix it and not the release manager.
>
> Having said that, if "during" the release process (i.e., cutting an RC)
> these backports cause delays for a release manager in getting the release
> out (e.g., CI flakiness introduced due to backports), release manager could
> be the ultimate arbiter on whether such a backport should be reverted or
> fixed by the committer/contributor. Hopefully such issues are caught much
> before a release process is started (e.g., CI running against release
> branches).
>
> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>
> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
> > whether a bug fix should be backported, and leave the decision to
> committer
> > (because they have the most context, and avoid a bottleneck in the
> > process).
> >
> > - Jie
> >
> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
> >
> > > My impression is that we have two opposing schools of thought here:
> > >
> > > 1. Backport as little as possible, to avoid unforeseen consequences
> > > 2. Backport as much as proves practical, to eliminate bugs in
> > > supported versions
> > >
> > > Do other people agree with this assessment?
> > >
> > > If so, how can we find common ground? One possible solution would be to
> > > leave the decision on backporting up to the committer, without
> > specifying a
> > > project-wide policy. This seems to be the status quo, and would lead to
> > > some variation across committers regarding what types of fixes are
> > > backported. We could also choose to delegate the decision to the
> release
> > > manager; I favor leaving the decision with the committer, to eliminate
> > the
> > > burden on release managers.
> > >
> > > Here's a thought: rather than defining a prescriptive "policy" that we
> > > expect committers to abide by, we could enumerate in the documentation
> > the
> > > competing concerns that we expect committers to consider when making
> > > decisions on backports. The committing docs could read something like:
> > >
> > > "When bug fixes are committed to master, the committer should evaluate
> > the
> > > fix to determine whether or not it should be backported to supported
> > > versions. This is left to the committer, but they are expected to weigh
> > the
> > > following concerns when making the decision:
> > >
> > > - Every backported change comes with a risk of unintended
> > > consequences. The change should be carefully evaluated to ensure
> that
> > such
> > > side-effects are highly unlikely.
> > > - As the complexity of applying a backport increases due to merge
> > > conflicts, the likelihood of unintended consequences also increases.
> > Bug
> > > fixes which require extensive rebasing should only be backported
> when
> > the
> > > bug is critical enough to warrant the risk.
> > > - Users of supported versions benefit greatly from the resolution of
> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
> > allayed
> > > for a given bug fix, it should be backported."
> > >
> > >
> > > Cheers,
> > > Greg
> > >
> > >
> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> > > wrote:
> > >
> > >> Back porting as little as possible is the ultimate goal for me. My
> > >> reasons are closely aligned with what Andrew wrote above.
> > >>
> > >> If we agree on this strategy, the next question is how to enforce it.
> My
> > >> intuition is that committers will lean towards back porting their
> > patches
> > >> in arguable cases, because humans tend to overestimate the importance
> of
> > >> their personal work. Delegating the decision in such cases to a
> release
> > >> manager in my opinion will help us enforce the strategy of minimal
> > number
> > >> backports. As a bonus, the release manager will have a much better
> > >> understanding of what's going on with the release, keyword: "more
> > >> ownership".
> > >>
> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> > >> andrew@schwartzmeyer.com> wrote:
> > >>
> > >>> I believe I fall somewhere between Alex and Ben.
> > >>>
> > >>> As for deciding what to backport or not, I lean toward Alex's view of
> > >>> backporting as little as possible (and agree with his criteria). My
> > >>> reasoning is that all changes can have unforeseen consequences,
> which I
> > >>> believe is something to be actively avoided in already released
> > versions.
> > >>> The reason for backporting patches to fix regressions is the same as
> > the
> > >>> reason to avoid backporting as much as possible: keep behavior
> > consistent
> > >>> (and safe) within a release. With that as the goal of a branch in
> > >>> maintenance mode, it makes sense to fix regressions, and make
> > exceptions to
> > >>> fix CVEs and other critical/blocking issues.
> > >>>
> > >>> As for who should decide what to backport, I lean toward Ben's view
> of
> > >>> the burden being on the committer. I don't think we should add more
> > work
> > >>> for release managers, and I think the committer/shepherd obviously
> has
> > the
> > >>> most understanding of the context around changes proposed for
> backport.
> > >>>
> > >>> Here's an example of a recent bugfix which I backported:
> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
> > >>>
> > >>> While normally I believe this change falls under "avoid due to
> > >>> unforeseen consequences," I made an exception as the bug was old,
> circa
> > >>> 2015, (indicating it had been an issue for others), and was causing
> > >>> recurring failures in testing. The fix itself was very small, meaning
> > it
> > >>> was easier to evaluate for possible side effects, so I felt a little
> > safer
> > >>> in that regard. The effect of not having the fix was a fatal and
> > undesired
> > >>> crash, which furthermore left troublesome side effects on the system
> > (you
> > >>> couldn't bring the agent back up). And lastly, a dependent project
> > (DC/OS)
> > >>> wanted it in their next bump, which necessitated backporting to the
> > release
> > >>> they were pulling in.
> > >>>
> > >>> I think in general we should backport only as necessary, and leave it
> > on
> > >>> the committers to decide if backporting a particular change is
> > necessary.
> > >>>
> > >>>
> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> > >>>
> > >>>> This is exactly where our views differ, Ben : )
> > >>>>
> > >>>> Ideally, I would like a release manager to have more ownership and
> > less
> > >>>> manual work. In my imagination, a release manager has more power and
> > >>>> control about dates, features, backports and everything that is
> > related
> > >>>> to
> > >>>> "their" branch. I would also like us to back port as little as
> > >>>> possible, to
> > >>>> simplify testing and releasing patch versions.
> > >>>>
> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
> bmahler@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> +user, I probably it would be good to hear from users as well.
> > >>>>>
> > >>>>> Please see the original proposal as well as Alex's proposal and let
> > us
> > >>>>> know
> > >>>>> your thoughts.
> > >>>>>
> > >>>>> To continue the discussion from where Alex left off:
> > >>>>>
> > >>>>> > Other bugs and significant improvements, e.g., performance, may
> be
> > >>>>> back
> > >>>>> ported,
> > >>>>> the release manager should ideally be the one who decides on this.
> > >>>>>
> > >>>>> I'm a little puzzled by this, why is the release manager involved?
> As
> > >>>>> we
> > >>>>> already document, backports occur when the bug is fixed, so this
> > >>>>> happens in
> > >>>>> the steady state of development, not at release time. The release
> > >>>>> manager
> > >>>>> only comes in at the time of the release itself, at which point all
> > >>>>> backports have already happened and the release manager handles the
> > >>>>> release
> > >>>>> process. Only blocker level issues can stop the release and while
> the
> > >>>>> release manager has a strong say, we should generally agree on what
> > >>>>> consists of a release blocking issue.
> > >>>>>
> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
> > >>>>> commit
> > >>>>> that applies cleanly, right after I commit it to master (with the
> > >>>>> exceptions I listed below).
> > >>>>>
> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
> alex@mesosphere.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>> > I would like to back port as little as possible. I suggest the
> > >>>>> following
> > >>>>> > criteria:
> > >>>>> >
> > >>>>> > * By default, regressions are back ported to existing release
> > >>>>> branches. A
> > >>>>> > bug is considered a regression if the functionality is present in
> > the
> > >>>>> > previous minor or patch version and is not affected by the bug
> > there.
> > >>>>> >
> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >>>>> >
> > >>>>> > * Other bugs and significant improvements, e.g., performance, may
> > be
> > >>>>> back
> > >>>>> > ported, the release manager should ideally be the one who decides
> > on
> > >>>>> this.
> > >>>>> >
> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
> vinodkone@apache.org
> > >
> > >>>>> wrote:
> > >>>>> >
> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
> > >>>>> points you
> > >>>>> > > made.
> > >>>>> > >
> > >>>>> > > Once we have consensus, would you mind updating the doc?
> > >>>>> > >
> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
> > >>>>> bmahler@apache.org>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > > > I realized recently that we aren't all on the same page with
> > >>>>> > backporting.
> > >>>>> > > > We currently only document the following:
> > >>>>> > > >
> > >>>>> > > > "Typically the fix for an issue that is affecting supported
> > >>>>> releases
> > >>>>> > > lands
> > >>>>> > > > on the master branch and is then backported to the release
> > >>>>> branch(es).
> > >>>>> > In
> > >>>>> > > > rare cases, the fix might directly go into a release branch
> > >>>>> without
> > >>>>> > > landing
> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)."
> [1]
> > >>>>> > > >
> > >>>>> > > > This leaves room for interpretation about what lies outside
> of
> > >>>>> > "typical".
> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
> I'd
> > >>>>> like
> > >>>>> to
> > >>>>> > > hear
> > >>>>> > > > what others have in mind:
> > >>>>> > > >
> > >>>>> > > > * By default, bug fixes at any level should be backported to
> > >>>>> existing
> > >>>>> > > > release branches if it affects those releases. Especially
> > >>>>> important:
> > >>>>> > > > crashes, bugs in non-experimental features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
> > >>>>> backport
> > >>>>> > > fixes
> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
> > >>>>> > experimental
> > >>>>> > > > features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional non-bug cases that can be backported:
> performance
> > >>>>> > > > improvements.
> > >>>>> > > >
> > >>>>> > > > I realize that there is a ton of subtlety here (even in terms
> > of
> > >>>>> which
> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
> > policy
> > >>>>> that
> > >>>>> > > gives
> > >>>>> > > > everyone the right mindset for common cases and then discuss
> > >>>>> corner
> > >>>>> > cases
> > >>>>> > > > on-demand in the future.
> > >>>>> > > >
> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>
> > >>
> > >
> >
>
Re: Backport Policy
Posted by Chun-Hung Hsiao <ch...@mesosphere.io>.
I just have a comment on a special case:
If a fix for a flaky test is easy to backport,
IMO we probably should backport it,
otherwise if someone backports another critical fix in the future,
it would take them extra effort to check all CI failures.
On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vi...@apache.org> wrote:
> I like how you summarized it Greg and I would vote for leaving the decision
> to the committer too. In addition to what others mentioned, I think
> committer should've the responsibility because if things break in a point
> release (after it is released), it is the committer and contributor who are
> on the hook to triage and fix it and not the release manager.
>
> Having said that, if "during" the release process (i.e., cutting an RC)
> these backports cause delays for a release manager in getting the release
> out (e.g., CI flakiness introduced due to backports), release manager could
> be the ultimate arbiter on whether such a backport should be reverted or
> fixed by the committer/contributor. Hopefully such issues are caught much
> before a release process is started (e.g., CI running against release
> branches).
>
> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
>
> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
> > whether a bug fix should be backported, and leave the decision to
> committer
> > (because they have the most context, and avoid a bottleneck in the
> > process).
> >
> > - Jie
> >
> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
> >
> > > My impression is that we have two opposing schools of thought here:
> > >
> > > 1. Backport as little as possible, to avoid unforeseen consequences
> > > 2. Backport as much as proves practical, to eliminate bugs in
> > > supported versions
> > >
> > > Do other people agree with this assessment?
> > >
> > > If so, how can we find common ground? One possible solution would be to
> > > leave the decision on backporting up to the committer, without
> > specifying a
> > > project-wide policy. This seems to be the status quo, and would lead to
> > > some variation across committers regarding what types of fixes are
> > > backported. We could also choose to delegate the decision to the
> release
> > > manager; I favor leaving the decision with the committer, to eliminate
> > the
> > > burden on release managers.
> > >
> > > Here's a thought: rather than defining a prescriptive "policy" that we
> > > expect committers to abide by, we could enumerate in the documentation
> > the
> > > competing concerns that we expect committers to consider when making
> > > decisions on backports. The committing docs could read something like:
> > >
> > > "When bug fixes are committed to master, the committer should evaluate
> > the
> > > fix to determine whether or not it should be backported to supported
> > > versions. This is left to the committer, but they are expected to weigh
> > the
> > > following concerns when making the decision:
> > >
> > > - Every backported change comes with a risk of unintended
> > > consequences. The change should be carefully evaluated to ensure
> that
> > such
> > > side-effects are highly unlikely.
> > > - As the complexity of applying a backport increases due to merge
> > > conflicts, the likelihood of unintended consequences also increases.
> > Bug
> > > fixes which require extensive rebasing should only be backported
> when
> > the
> > > bug is critical enough to warrant the risk.
> > > - Users of supported versions benefit greatly from the resolution of
> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be
> > allayed
> > > for a given bug fix, it should be backported."
> > >
> > >
> > > Cheers,
> > > Greg
> > >
> > >
> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> > > wrote:
> > >
> > >> Back porting as little as possible is the ultimate goal for me. My
> > >> reasons are closely aligned with what Andrew wrote above.
> > >>
> > >> If we agree on this strategy, the next question is how to enforce it.
> My
> > >> intuition is that committers will lean towards back porting their
> > patches
> > >> in arguable cases, because humans tend to overestimate the importance
> of
> > >> their personal work. Delegating the decision in such cases to a
> release
> > >> manager in my opinion will help us enforce the strategy of minimal
> > number
> > >> backports. As a bonus, the release manager will have a much better
> > >> understanding of what's going on with the release, keyword: "more
> > >> ownership".
> > >>
> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> > >> andrew@schwartzmeyer.com> wrote:
> > >>
> > >>> I believe I fall somewhere between Alex and Ben.
> > >>>
> > >>> As for deciding what to backport or not, I lean toward Alex's view of
> > >>> backporting as little as possible (and agree with his criteria). My
> > >>> reasoning is that all changes can have unforeseen consequences,
> which I
> > >>> believe is something to be actively avoided in already released
> > versions.
> > >>> The reason for backporting patches to fix regressions is the same as
> > the
> > >>> reason to avoid backporting as much as possible: keep behavior
> > consistent
> > >>> (and safe) within a release. With that as the goal of a branch in
> > >>> maintenance mode, it makes sense to fix regressions, and make
> > exceptions to
> > >>> fix CVEs and other critical/blocking issues.
> > >>>
> > >>> As for who should decide what to backport, I lean toward Ben's view
> of
> > >>> the burden being on the committer. I don't think we should add more
> > work
> > >>> for release managers, and I think the committer/shepherd obviously
> has
> > the
> > >>> most understanding of the context around changes proposed for
> backport.
> > >>>
> > >>> Here's an example of a recent bugfix which I backported:
> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
> > >>>
> > >>> While normally I believe this change falls under "avoid due to
> > >>> unforeseen consequences," I made an exception as the bug was old,
> circa
> > >>> 2015, (indicating it had been an issue for others), and was causing
> > >>> recurring failures in testing. The fix itself was very small, meaning
> > it
> > >>> was easier to evaluate for possible side effects, so I felt a little
> > safer
> > >>> in that regard. The effect of not having the fix was a fatal and
> > undesired
> > >>> crash, which furthermore left troublesome side effects on the system
> > (you
> > >>> couldn't bring the agent back up). And lastly, a dependent project
> > (DC/OS)
> > >>> wanted it in their next bump, which necessitated backporting to the
> > release
> > >>> they were pulling in.
> > >>>
> > >>> I think in general we should backport only as necessary, and leave it
> > on
> > >>> the committers to decide if backporting a particular change is
> > necessary.
> > >>>
> > >>>
> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> > >>>
> > >>>> This is exactly where our views differ, Ben : )
> > >>>>
> > >>>> Ideally, I would like a release manager to have more ownership and
> > less
> > >>>> manual work. In my imagination, a release manager has more power and
> > >>>> control about dates, features, backports and everything that is
> > related
> > >>>> to
> > >>>> "their" branch. I would also like us to back port as little as
> > >>>> possible, to
> > >>>> simplify testing and releasing patch versions.
> > >>>>
> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
> bmahler@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> +user, I probably it would be good to hear from users as well.
> > >>>>>
> > >>>>> Please see the original proposal as well as Alex's proposal and let
> > us
> > >>>>> know
> > >>>>> your thoughts.
> > >>>>>
> > >>>>> To continue the discussion from where Alex left off:
> > >>>>>
> > >>>>> > Other bugs and significant improvements, e.g., performance, may
> be
> > >>>>> back
> > >>>>> ported,
> > >>>>> the release manager should ideally be the one who decides on this.
> > >>>>>
> > >>>>> I'm a little puzzled by this, why is the release manager involved?
> As
> > >>>>> we
> > >>>>> already document, backports occur when the bug is fixed, so this
> > >>>>> happens in
> > >>>>> the steady state of development, not at release time. The release
> > >>>>> manager
> > >>>>> only comes in at the time of the release itself, at which point all
> > >>>>> backports have already happened and the release manager handles the
> > >>>>> release
> > >>>>> process. Only blocker level issues can stop the release and while
> the
> > >>>>> release manager has a strong say, we should generally agree on what
> > >>>>> consists of a release blocking issue.
> > >>>>>
> > >>>>> Just to clarify my workflow, I generally backport every bug fix I
> > >>>>> commit
> > >>>>> that applies cleanly, right after I commit it to master (with the
> > >>>>> exceptions I listed below).
> > >>>>>
> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
> alex@mesosphere.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>> > I would like to back port as little as possible. I suggest the
> > >>>>> following
> > >>>>> > criteria:
> > >>>>> >
> > >>>>> > * By default, regressions are back ported to existing release
> > >>>>> branches. A
> > >>>>> > bug is considered a regression if the functionality is present in
> > the
> > >>>>> > previous minor or patch version and is not affected by the bug
> > there.
> > >>>>> >
> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >>>>> >
> > >>>>> > * Other bugs and significant improvements, e.g., performance, may
> > be
> > >>>>> back
> > >>>>> > ported, the release manager should ideally be the one who decides
> > on
> > >>>>> this.
> > >>>>> >
> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
> vinodkone@apache.org
> > >
> > >>>>> wrote:
> > >>>>> >
> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
> > >>>>> points you
> > >>>>> > > made.
> > >>>>> > >
> > >>>>> > > Once we have consensus, would you mind updating the doc?
> > >>>>> > >
> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
> > >>>>> bmahler@apache.org>
> > >>>>> > > wrote:
> > >>>>> > >
> > >>>>> > > > I realized recently that we aren't all on the same page with
> > >>>>> > backporting.
> > >>>>> > > > We currently only document the following:
> > >>>>> > > >
> > >>>>> > > > "Typically the fix for an issue that is affecting supported
> > >>>>> releases
> > >>>>> > > lands
> > >>>>> > > > on the master branch and is then backported to the release
> > >>>>> branch(es).
> > >>>>> > In
> > >>>>> > > > rare cases, the fix might directly go into a release branch
> > >>>>> without
> > >>>>> > > landing
> > >>>>> > > > on master (e.g., fix / issue is not applicable to master)."
> [1]
> > >>>>> > > >
> > >>>>> > > > This leaves room for interpretation about what lies outside
> of
> > >>>>> > "typical".
> > >>>>> > > > Here's the simplest way I can explain what I stick to, and
> I'd
> > >>>>> like
> > >>>>> to
> > >>>>> > > hear
> > >>>>> > > > what others have in mind:
> > >>>>> > > >
> > >>>>> > > > * By default, bug fixes at any level should be backported to
> > >>>>> existing
> > >>>>> > > > release branches if it affects those releases. Especially
> > >>>>> important:
> > >>>>> > > > crashes, bugs in non-experimental features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional cases that can omit backporting: difficult to
> > >>>>> backport
> > >>>>> > > fixes
> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
> > >>>>> > experimental
> > >>>>> > > > features.
> > >>>>> > > >
> > >>>>> > > > * Exceptional non-bug cases that can be backported:
> performance
> > >>>>> > > > improvements.
> > >>>>> > > >
> > >>>>> > > > I realize that there is a ton of subtlety here (even in terms
> > of
> > >>>>> which
> > >>>>> > > > things are defined as bugs). But I hope we can lay down a
> > policy
> > >>>>> that
> > >>>>> > > gives
> > >>>>> > > > everyone the right mindset for common cases and then discuss
> > >>>>> corner
> > >>>>> > cases
> > >>>>> > > > on-demand in the future.
> > >>>>> > > >
> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>
> > >>
> > >
> >
>
Re: Backport Policy
Posted by Vinod Kone <vi...@apache.org>.
I like how you summarized it Greg and I would vote for leaving the decision
to the committer too. In addition to what others mentioned, I think
committer should've the responsibility because if things break in a point
release (after it is released), it is the committer and contributor who are
on the hook to triage and fix it and not the release manager.
Having said that, if "during" the release process (i.e., cutting an RC)
these backports cause delays for a release manager in getting the release
out (e.g., CI flakiness introduced due to backports), release manager could
be the ultimate arbiter on whether such a backport should be reverted or
fixed by the committer/contributor. Hopefully such issues are caught much
before a release process is started (e.g., CI running against release
branches).
On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
> Greg, I like your idea of adding a prescriptive "policy" when evaluating
> whether a bug fix should be backported, and leave the decision to committer
> (because they have the most context, and avoid a bottleneck in the
> process).
>
> - Jie
>
> On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
>
> > My impression is that we have two opposing schools of thought here:
> >
> > 1. Backport as little as possible, to avoid unforeseen consequences
> > 2. Backport as much as proves practical, to eliminate bugs in
> > supported versions
> >
> > Do other people agree with this assessment?
> >
> > If so, how can we find common ground? One possible solution would be to
> > leave the decision on backporting up to the committer, without
> specifying a
> > project-wide policy. This seems to be the status quo, and would lead to
> > some variation across committers regarding what types of fixes are
> > backported. We could also choose to delegate the decision to the release
> > manager; I favor leaving the decision with the committer, to eliminate
> the
> > burden on release managers.
> >
> > Here's a thought: rather than defining a prescriptive "policy" that we
> > expect committers to abide by, we could enumerate in the documentation
> the
> > competing concerns that we expect committers to consider when making
> > decisions on backports. The committing docs could read something like:
> >
> > "When bug fixes are committed to master, the committer should evaluate
> the
> > fix to determine whether or not it should be backported to supported
> > versions. This is left to the committer, but they are expected to weigh
> the
> > following concerns when making the decision:
> >
> > - Every backported change comes with a risk of unintended
> > consequences. The change should be carefully evaluated to ensure that
> such
> > side-effects are highly unlikely.
> > - As the complexity of applying a backport increases due to merge
> > conflicts, the likelihood of unintended consequences also increases.
> Bug
> > fixes which require extensive rebasing should only be backported when
> the
> > bug is critical enough to warrant the risk.
> > - Users of supported versions benefit greatly from the resolution of
> > bugs in point releases. Thus, whenever concerns #1 and #2 can be
> allayed
> > for a given bug fix, it should be backported."
> >
> >
> > Cheers,
> > Greg
> >
> >
> > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> > wrote:
> >
> >> Back porting as little as possible is the ultimate goal for me. My
> >> reasons are closely aligned with what Andrew wrote above.
> >>
> >> If we agree on this strategy, the next question is how to enforce it. My
> >> intuition is that committers will lean towards back porting their
> patches
> >> in arguable cases, because humans tend to overestimate the importance of
> >> their personal work. Delegating the decision in such cases to a release
> >> manager in my opinion will help us enforce the strategy of minimal
> number
> >> backports. As a bonus, the release manager will have a much better
> >> understanding of what's going on with the release, keyword: "more
> >> ownership".
> >>
> >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> >> andrew@schwartzmeyer.com> wrote:
> >>
> >>> I believe I fall somewhere between Alex and Ben.
> >>>
> >>> As for deciding what to backport or not, I lean toward Alex's view of
> >>> backporting as little as possible (and agree with his criteria). My
> >>> reasoning is that all changes can have unforeseen consequences, which I
> >>> believe is something to be actively avoided in already released
> versions.
> >>> The reason for backporting patches to fix regressions is the same as
> the
> >>> reason to avoid backporting as much as possible: keep behavior
> consistent
> >>> (and safe) within a release. With that as the goal of a branch in
> >>> maintenance mode, it makes sense to fix regressions, and make
> exceptions to
> >>> fix CVEs and other critical/blocking issues.
> >>>
> >>> As for who should decide what to backport, I lean toward Ben's view of
> >>> the burden being on the committer. I don't think we should add more
> work
> >>> for release managers, and I think the committer/shepherd obviously has
> the
> >>> most understanding of the context around changes proposed for backport.
> >>>
> >>> Here's an example of a recent bugfix which I backported:
> >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
> >>>
> >>> While normally I believe this change falls under "avoid due to
> >>> unforeseen consequences," I made an exception as the bug was old, circa
> >>> 2015, (indicating it had been an issue for others), and was causing
> >>> recurring failures in testing. The fix itself was very small, meaning
> it
> >>> was easier to evaluate for possible side effects, so I felt a little
> safer
> >>> in that regard. The effect of not having the fix was a fatal and
> undesired
> >>> crash, which furthermore left troublesome side effects on the system
> (you
> >>> couldn't bring the agent back up). And lastly, a dependent project
> (DC/OS)
> >>> wanted it in their next bump, which necessitated backporting to the
> release
> >>> they were pulling in.
> >>>
> >>> I think in general we should backport only as necessary, and leave it
> on
> >>> the committers to decide if backporting a particular change is
> necessary.
> >>>
> >>>
> >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> >>>
> >>>> This is exactly where our views differ, Ben : )
> >>>>
> >>>> Ideally, I would like a release manager to have more ownership and
> less
> >>>> manual work. In my imagination, a release manager has more power and
> >>>> control about dates, features, backports and everything that is
> related
> >>>> to
> >>>> "their" branch. I would also like us to back port as little as
> >>>> possible, to
> >>>> simplify testing and releasing patch versions.
> >>>>
> >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> >>>> wrote:
> >>>>
> >>>> +user, I probably it would be good to hear from users as well.
> >>>>>
> >>>>> Please see the original proposal as well as Alex's proposal and let
> us
> >>>>> know
> >>>>> your thoughts.
> >>>>>
> >>>>> To continue the discussion from where Alex left off:
> >>>>>
> >>>>> > Other bugs and significant improvements, e.g., performance, may be
> >>>>> back
> >>>>> ported,
> >>>>> the release manager should ideally be the one who decides on this.
> >>>>>
> >>>>> I'm a little puzzled by this, why is the release manager involved? As
> >>>>> we
> >>>>> already document, backports occur when the bug is fixed, so this
> >>>>> happens in
> >>>>> the steady state of development, not at release time. The release
> >>>>> manager
> >>>>> only comes in at the time of the release itself, at which point all
> >>>>> backports have already happened and the release manager handles the
> >>>>> release
> >>>>> process. Only blocker level issues can stop the release and while the
> >>>>> release manager has a strong say, we should generally agree on what
> >>>>> consists of a release blocking issue.
> >>>>>
> >>>>> Just to clarify my workflow, I generally backport every bug fix I
> >>>>> commit
> >>>>> that applies cleanly, right after I commit it to master (with the
> >>>>> exceptions I listed below).
> >>>>>
> >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <alex@mesosphere.com
> >
> >>>>> wrote:
> >>>>>
> >>>>> > I would like to back port as little as possible. I suggest the
> >>>>> following
> >>>>> > criteria:
> >>>>> >
> >>>>> > * By default, regressions are back ported to existing release
> >>>>> branches. A
> >>>>> > bug is considered a regression if the functionality is present in
> the
> >>>>> > previous minor or patch version and is not affected by the bug
> there.
> >>>>> >
> >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> >>>>> >
> >>>>> > * Other bugs and significant improvements, e.g., performance, may
> be
> >>>>> back
> >>>>> > ported, the release manager should ideally be the one who decides
> on
> >>>>> this.
> >>>>> >
> >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vinodkone@apache.org
> >
> >>>>> wrote:
> >>>>> >
> >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
> >>>>> points you
> >>>>> > > made.
> >>>>> > >
> >>>>> > > Once we have consensus, would you mind updating the doc?
> >>>>> > >
> >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
> >>>>> bmahler@apache.org>
> >>>>> > > wrote:
> >>>>> > >
> >>>>> > > > I realized recently that we aren't all on the same page with
> >>>>> > backporting.
> >>>>> > > > We currently only document the following:
> >>>>> > > >
> >>>>> > > > "Typically the fix for an issue that is affecting supported
> >>>>> releases
> >>>>> > > lands
> >>>>> > > > on the master branch and is then backported to the release
> >>>>> branch(es).
> >>>>> > In
> >>>>> > > > rare cases, the fix might directly go into a release branch
> >>>>> without
> >>>>> > > landing
> >>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> >>>>> > > >
> >>>>> > > > This leaves room for interpretation about what lies outside of
> >>>>> > "typical".
> >>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
> >>>>> like
> >>>>> to
> >>>>> > > hear
> >>>>> > > > what others have in mind:
> >>>>> > > >
> >>>>> > > > * By default, bug fixes at any level should be backported to
> >>>>> existing
> >>>>> > > > release branches if it affects those releases. Especially
> >>>>> important:
> >>>>> > > > crashes, bugs in non-experimental features.
> >>>>> > > >
> >>>>> > > > * Exceptional cases that can omit backporting: difficult to
> >>>>> backport
> >>>>> > > fixes
> >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
> >>>>> > experimental
> >>>>> > > > features.
> >>>>> > > >
> >>>>> > > > * Exceptional non-bug cases that can be backported: performance
> >>>>> > > > improvements.
> >>>>> > > >
> >>>>> > > > I realize that there is a ton of subtlety here (even in terms
> of
> >>>>> which
> >>>>> > > > things are defined as bugs). But I hope we can lay down a
> policy
> >>>>> that
> >>>>> > > gives
> >>>>> > > > everyone the right mindset for common cases and then discuss
> >>>>> corner
> >>>>> > cases
> >>>>> > > > on-demand in the future.
> >>>>> > > >
> >>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> >>>>>
> >>>
> >>
> >
>
Re: Backport Policy
Posted by Vinod Kone <vi...@apache.org>.
I like how you summarized it Greg and I would vote for leaving the decision
to the committer too. In addition to what others mentioned, I think
committer should've the responsibility because if things break in a point
release (after it is released), it is the committer and contributor who are
on the hook to triage and fix it and not the release manager.
Having said that, if "during" the release process (i.e., cutting an RC)
these backports cause delays for a release manager in getting the release
out (e.g., CI flakiness introduced due to backports), release manager could
be the ultimate arbiter on whether such a backport should be reverted or
fixed by the committer/contributor. Hopefully such issues are caught much
before a release process is started (e.g., CI running against release
branches).
On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yu...@gmail.com> wrote:
> Greg, I like your idea of adding a prescriptive "policy" when evaluating
> whether a bug fix should be backported, and leave the decision to committer
> (because they have the most context, and avoid a bottleneck in the
> process).
>
> - Jie
>
> On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
>
> > My impression is that we have two opposing schools of thought here:
> >
> > 1. Backport as little as possible, to avoid unforeseen consequences
> > 2. Backport as much as proves practical, to eliminate bugs in
> > supported versions
> >
> > Do other people agree with this assessment?
> >
> > If so, how can we find common ground? One possible solution would be to
> > leave the decision on backporting up to the committer, without
> specifying a
> > project-wide policy. This seems to be the status quo, and would lead to
> > some variation across committers regarding what types of fixes are
> > backported. We could also choose to delegate the decision to the release
> > manager; I favor leaving the decision with the committer, to eliminate
> the
> > burden on release managers.
> >
> > Here's a thought: rather than defining a prescriptive "policy" that we
> > expect committers to abide by, we could enumerate in the documentation
> the
> > competing concerns that we expect committers to consider when making
> > decisions on backports. The committing docs could read something like:
> >
> > "When bug fixes are committed to master, the committer should evaluate
> the
> > fix to determine whether or not it should be backported to supported
> > versions. This is left to the committer, but they are expected to weigh
> the
> > following concerns when making the decision:
> >
> > - Every backported change comes with a risk of unintended
> > consequences. The change should be carefully evaluated to ensure that
> such
> > side-effects are highly unlikely.
> > - As the complexity of applying a backport increases due to merge
> > conflicts, the likelihood of unintended consequences also increases.
> Bug
> > fixes which require extensive rebasing should only be backported when
> the
> > bug is critical enough to warrant the risk.
> > - Users of supported versions benefit greatly from the resolution of
> > bugs in point releases. Thus, whenever concerns #1 and #2 can be
> allayed
> > for a given bug fix, it should be backported."
> >
> >
> > Cheers,
> > Greg
> >
> >
> > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> > wrote:
> >
> >> Back porting as little as possible is the ultimate goal for me. My
> >> reasons are closely aligned with what Andrew wrote above.
> >>
> >> If we agree on this strategy, the next question is how to enforce it. My
> >> intuition is that committers will lean towards back porting their
> patches
> >> in arguable cases, because humans tend to overestimate the importance of
> >> their personal work. Delegating the decision in such cases to a release
> >> manager in my opinion will help us enforce the strategy of minimal
> number
> >> backports. As a bonus, the release manager will have a much better
> >> understanding of what's going on with the release, keyword: "more
> >> ownership".
> >>
> >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> >> andrew@schwartzmeyer.com> wrote:
> >>
> >>> I believe I fall somewhere between Alex and Ben.
> >>>
> >>> As for deciding what to backport or not, I lean toward Alex's view of
> >>> backporting as little as possible (and agree with his criteria). My
> >>> reasoning is that all changes can have unforeseen consequences, which I
> >>> believe is something to be actively avoided in already released
> versions.
> >>> The reason for backporting patches to fix regressions is the same as
> the
> >>> reason to avoid backporting as much as possible: keep behavior
> consistent
> >>> (and safe) within a release. With that as the goal of a branch in
> >>> maintenance mode, it makes sense to fix regressions, and make
> exceptions to
> >>> fix CVEs and other critical/blocking issues.
> >>>
> >>> As for who should decide what to backport, I lean toward Ben's view of
> >>> the burden being on the committer. I don't think we should add more
> work
> >>> for release managers, and I think the committer/shepherd obviously has
> the
> >>> most understanding of the context around changes proposed for backport.
> >>>
> >>> Here's an example of a recent bugfix which I backported:
> >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
> >>>
> >>> While normally I believe this change falls under "avoid due to
> >>> unforeseen consequences," I made an exception as the bug was old, circa
> >>> 2015, (indicating it had been an issue for others), and was causing
> >>> recurring failures in testing. The fix itself was very small, meaning
> it
> >>> was easier to evaluate for possible side effects, so I felt a little
> safer
> >>> in that regard. The effect of not having the fix was a fatal and
> undesired
> >>> crash, which furthermore left troublesome side effects on the system
> (you
> >>> couldn't bring the agent back up). And lastly, a dependent project
> (DC/OS)
> >>> wanted it in their next bump, which necessitated backporting to the
> release
> >>> they were pulling in.
> >>>
> >>> I think in general we should backport only as necessary, and leave it
> on
> >>> the committers to decide if backporting a particular change is
> necessary.
> >>>
> >>>
> >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> >>>
> >>>> This is exactly where our views differ, Ben : )
> >>>>
> >>>> Ideally, I would like a release manager to have more ownership and
> less
> >>>> manual work. In my imagination, a release manager has more power and
> >>>> control about dates, features, backports and everything that is
> related
> >>>> to
> >>>> "their" branch. I would also like us to back port as little as
> >>>> possible, to
> >>>> simplify testing and releasing patch versions.
> >>>>
> >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> >>>> wrote:
> >>>>
> >>>> +user, I probably it would be good to hear from users as well.
> >>>>>
> >>>>> Please see the original proposal as well as Alex's proposal and let
> us
> >>>>> know
> >>>>> your thoughts.
> >>>>>
> >>>>> To continue the discussion from where Alex left off:
> >>>>>
> >>>>> > Other bugs and significant improvements, e.g., performance, may be
> >>>>> back
> >>>>> ported,
> >>>>> the release manager should ideally be the one who decides on this.
> >>>>>
> >>>>> I'm a little puzzled by this, why is the release manager involved? As
> >>>>> we
> >>>>> already document, backports occur when the bug is fixed, so this
> >>>>> happens in
> >>>>> the steady state of development, not at release time. The release
> >>>>> manager
> >>>>> only comes in at the time of the release itself, at which point all
> >>>>> backports have already happened and the release manager handles the
> >>>>> release
> >>>>> process. Only blocker level issues can stop the release and while the
> >>>>> release manager has a strong say, we should generally agree on what
> >>>>> consists of a release blocking issue.
> >>>>>
> >>>>> Just to clarify my workflow, I generally backport every bug fix I
> >>>>> commit
> >>>>> that applies cleanly, right after I commit it to master (with the
> >>>>> exceptions I listed below).
> >>>>>
> >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <alex@mesosphere.com
> >
> >>>>> wrote:
> >>>>>
> >>>>> > I would like to back port as little as possible. I suggest the
> >>>>> following
> >>>>> > criteria:
> >>>>> >
> >>>>> > * By default, regressions are back ported to existing release
> >>>>> branches. A
> >>>>> > bug is considered a regression if the functionality is present in
> the
> >>>>> > previous minor or patch version and is not affected by the bug
> there.
> >>>>> >
> >>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> >>>>> >
> >>>>> > * Other bugs and significant improvements, e.g., performance, may
> be
> >>>>> back
> >>>>> > ported, the release manager should ideally be the one who decides
> on
> >>>>> this.
> >>>>> >
> >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vinodkone@apache.org
> >
> >>>>> wrote:
> >>>>> >
> >>>>> > > Ben, thanks for the clarification. I'm in agreement with the
> >>>>> points you
> >>>>> > > made.
> >>>>> > >
> >>>>> > > Once we have consensus, would you mind updating the doc?
> >>>>> > >
> >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
> >>>>> bmahler@apache.org>
> >>>>> > > wrote:
> >>>>> > >
> >>>>> > > > I realized recently that we aren't all on the same page with
> >>>>> > backporting.
> >>>>> > > > We currently only document the following:
> >>>>> > > >
> >>>>> > > > "Typically the fix for an issue that is affecting supported
> >>>>> releases
> >>>>> > > lands
> >>>>> > > > on the master branch and is then backported to the release
> >>>>> branch(es).
> >>>>> > In
> >>>>> > > > rare cases, the fix might directly go into a release branch
> >>>>> without
> >>>>> > > landing
> >>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> >>>>> > > >
> >>>>> > > > This leaves room for interpretation about what lies outside of
> >>>>> > "typical".
> >>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
> >>>>> like
> >>>>> to
> >>>>> > > hear
> >>>>> > > > what others have in mind:
> >>>>> > > >
> >>>>> > > > * By default, bug fixes at any level should be backported to
> >>>>> existing
> >>>>> > > > release branches if it affects those releases. Especially
> >>>>> important:
> >>>>> > > > crashes, bugs in non-experimental features.
> >>>>> > > >
> >>>>> > > > * Exceptional cases that can omit backporting: difficult to
> >>>>> backport
> >>>>> > > fixes
> >>>>> > > > (especially if the bugs are deemed of low priority), bugs in
> >>>>> > experimental
> >>>>> > > > features.
> >>>>> > > >
> >>>>> > > > * Exceptional non-bug cases that can be backported: performance
> >>>>> > > > improvements.
> >>>>> > > >
> >>>>> > > > I realize that there is a ton of subtlety here (even in terms
> of
> >>>>> which
> >>>>> > > > things are defined as bugs). But I hope we can lay down a
> policy
> >>>>> that
> >>>>> > > gives
> >>>>> > > > everyone the right mindset for common cases and then discuss
> >>>>> corner
> >>>>> > cases
> >>>>> > > > on-demand in the future.
> >>>>> > > >
> >>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> >>>>>
> >>>
> >>
> >
>
Re: Backport Policy
Posted by Jie Yu <yu...@gmail.com>.
Greg, I like your idea of adding a prescriptive "policy" when evaluating
whether a bug fix should be backported, and leave the decision to committer
(because they have the most context, and avoid a bottleneck in the process).
- Jie
On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
> My impression is that we have two opposing schools of thought here:
>
> 1. Backport as little as possible, to avoid unforeseen consequences
> 2. Backport as much as proves practical, to eliminate bugs in
> supported versions
>
> Do other people agree with this assessment?
>
> If so, how can we find common ground? One possible solution would be to
> leave the decision on backporting up to the committer, without specifying a
> project-wide policy. This seems to be the status quo, and would lead to
> some variation across committers regarding what types of fixes are
> backported. We could also choose to delegate the decision to the release
> manager; I favor leaving the decision with the committer, to eliminate the
> burden on release managers.
>
> Here's a thought: rather than defining a prescriptive "policy" that we
> expect committers to abide by, we could enumerate in the documentation the
> competing concerns that we expect committers to consider when making
> decisions on backports. The committing docs could read something like:
>
> "When bug fixes are committed to master, the committer should evaluate the
> fix to determine whether or not it should be backported to supported
> versions. This is left to the committer, but they are expected to weigh the
> following concerns when making the decision:
>
> - Every backported change comes with a risk of unintended
> consequences. The change should be carefully evaluated to ensure that such
> side-effects are highly unlikely.
> - As the complexity of applying a backport increases due to merge
> conflicts, the likelihood of unintended consequences also increases. Bug
> fixes which require extensive rebasing should only be backported when the
> bug is critical enough to warrant the risk.
> - Users of supported versions benefit greatly from the resolution of
> bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
> for a given bug fix, it should be backported."
>
>
> Cheers,
> Greg
>
>
> On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
>> Back porting as little as possible is the ultimate goal for me. My
>> reasons are closely aligned with what Andrew wrote above.
>>
>> If we agree on this strategy, the next question is how to enforce it. My
>> intuition is that committers will lean towards back porting their patches
>> in arguable cases, because humans tend to overestimate the importance of
>> their personal work. Delegating the decision in such cases to a release
>> manager in my opinion will help us enforce the strategy of minimal number
>> backports. As a bonus, the release manager will have a much better
>> understanding of what's going on with the release, keyword: "more
>> ownership".
>>
>> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> andrew@schwartzmeyer.com> wrote:
>>
>>> I believe I fall somewhere between Alex and Ben.
>>>
>>> As for deciding what to backport or not, I lean toward Alex's view of
>>> backporting as little as possible (and agree with his criteria). My
>>> reasoning is that all changes can have unforeseen consequences, which I
>>> believe is something to be actively avoided in already released versions.
>>> The reason for backporting patches to fix regressions is the same as the
>>> reason to avoid backporting as much as possible: keep behavior consistent
>>> (and safe) within a release. With that as the goal of a branch in
>>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>>> fix CVEs and other critical/blocking issues.
>>>
>>> As for who should decide what to backport, I lean toward Ben's view of
>>> the burden being on the committer. I don't think we should add more work
>>> for release managers, and I think the committer/shepherd obviously has the
>>> most understanding of the context around changes proposed for backport.
>>>
>>> Here's an example of a recent bugfix which I backported:
>>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>>
>>> While normally I believe this change falls under "avoid due to
>>> unforeseen consequences," I made an exception as the bug was old, circa
>>> 2015, (indicating it had been an issue for others), and was causing
>>> recurring failures in testing. The fix itself was very small, meaning it
>>> was easier to evaluate for possible side effects, so I felt a little safer
>>> in that regard. The effect of not having the fix was a fatal and undesired
>>> crash, which furthermore left troublesome side effects on the system (you
>>> couldn't bring the agent back up). And lastly, a dependent project (DC/OS)
>>> wanted it in their next bump, which necessitated backporting to the release
>>> they were pulling in.
>>>
>>> I think in general we should backport only as necessary, and leave it on
>>> the committers to decide if backporting a particular change is necessary.
>>>
>>>
>>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>>
>>>> This is exactly where our views differ, Ben : )
>>>>
>>>> Ideally, I would like a release manager to have more ownership and less
>>>> manual work. In my imagination, a release manager has more power and
>>>> control about dates, features, backports and everything that is related
>>>> to
>>>> "their" branch. I would also like us to back port as little as
>>>> possible, to
>>>> simplify testing and releasing patch versions.
>>>>
>>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>>>> wrote:
>>>>
>>>> +user, I probably it would be good to hear from users as well.
>>>>>
>>>>> Please see the original proposal as well as Alex's proposal and let us
>>>>> know
>>>>> your thoughts.
>>>>>
>>>>> To continue the discussion from where Alex left off:
>>>>>
>>>>> > Other bugs and significant improvements, e.g., performance, may be
>>>>> back
>>>>> ported,
>>>>> the release manager should ideally be the one who decides on this.
>>>>>
>>>>> I'm a little puzzled by this, why is the release manager involved? As
>>>>> we
>>>>> already document, backports occur when the bug is fixed, so this
>>>>> happens in
>>>>> the steady state of development, not at release time. The release
>>>>> manager
>>>>> only comes in at the time of the release itself, at which point all
>>>>> backports have already happened and the release manager handles the
>>>>> release
>>>>> process. Only blocker level issues can stop the release and while the
>>>>> release manager has a strong say, we should generally agree on what
>>>>> consists of a release blocking issue.
>>>>>
>>>>> Just to clarify my workflow, I generally backport every bug fix I
>>>>> commit
>>>>> that applies cleanly, right after I commit it to master (with the
>>>>> exceptions I listed below).
>>>>>
>>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>>>> wrote:
>>>>>
>>>>> > I would like to back port as little as possible. I suggest the
>>>>> following
>>>>> > criteria:
>>>>> >
>>>>> > * By default, regressions are back ported to existing release
>>>>> branches. A
>>>>> > bug is considered a regression if the functionality is present in the
>>>>> > previous minor or patch version and is not affected by the bug there.
>>>>> >
>>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>>>> >
>>>>> > * Other bugs and significant improvements, e.g., performance, may be
>>>>> back
>>>>> > ported, the release manager should ideally be the one who decides on
>>>>> this.
>>>>> >
>>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>>>> points you
>>>>> > > made.
>>>>> > >
>>>>> > > Once we have consensus, would you mind updating the doc?
>>>>> > >
>>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>>>> bmahler@apache.org>
>>>>> > > wrote:
>>>>> > >
>>>>> > > > I realized recently that we aren't all on the same page with
>>>>> > backporting.
>>>>> > > > We currently only document the following:
>>>>> > > >
>>>>> > > > "Typically the fix for an issue that is affecting supported
>>>>> releases
>>>>> > > lands
>>>>> > > > on the master branch and is then backported to the release
>>>>> branch(es).
>>>>> > In
>>>>> > > > rare cases, the fix might directly go into a release branch
>>>>> without
>>>>> > > landing
>>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>>>> > > >
>>>>> > > > This leaves room for interpretation about what lies outside of
>>>>> > "typical".
>>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
>>>>> like
>>>>> to
>>>>> > > hear
>>>>> > > > what others have in mind:
>>>>> > > >
>>>>> > > > * By default, bug fixes at any level should be backported to
>>>>> existing
>>>>> > > > release branches if it affects those releases. Especially
>>>>> important:
>>>>> > > > crashes, bugs in non-experimental features.
>>>>> > > >
>>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>>>> backport
>>>>> > > fixes
>>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>>>> > experimental
>>>>> > > > features.
>>>>> > > >
>>>>> > > > * Exceptional non-bug cases that can be backported: performance
>>>>> > > > improvements.
>>>>> > > >
>>>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>>>> which
>>>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>>>> that
>>>>> > > gives
>>>>> > > > everyone the right mindset for common cases and then discuss
>>>>> corner
>>>>> > cases
>>>>> > > > on-demand in the future.
>>>>> > > >
>>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>>
>>>
>>
>
Re: Backport Policy
Posted by Jie Yu <yu...@gmail.com>.
Greg, I like your idea of adding a prescriptive "policy" when evaluating
whether a bug fix should be backported, and leave the decision to committer
(because they have the most context, and avoid a bottleneck in the process).
- Jie
On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <gr...@mesosphere.io> wrote:
> My impression is that we have two opposing schools of thought here:
>
> 1. Backport as little as possible, to avoid unforeseen consequences
> 2. Backport as much as proves practical, to eliminate bugs in
> supported versions
>
> Do other people agree with this assessment?
>
> If so, how can we find common ground? One possible solution would be to
> leave the decision on backporting up to the committer, without specifying a
> project-wide policy. This seems to be the status quo, and would lead to
> some variation across committers regarding what types of fixes are
> backported. We could also choose to delegate the decision to the release
> manager; I favor leaving the decision with the committer, to eliminate the
> burden on release managers.
>
> Here's a thought: rather than defining a prescriptive "policy" that we
> expect committers to abide by, we could enumerate in the documentation the
> competing concerns that we expect committers to consider when making
> decisions on backports. The committing docs could read something like:
>
> "When bug fixes are committed to master, the committer should evaluate the
> fix to determine whether or not it should be backported to supported
> versions. This is left to the committer, but they are expected to weigh the
> following concerns when making the decision:
>
> - Every backported change comes with a risk of unintended
> consequences. The change should be carefully evaluated to ensure that such
> side-effects are highly unlikely.
> - As the complexity of applying a backport increases due to merge
> conflicts, the likelihood of unintended consequences also increases. Bug
> fixes which require extensive rebasing should only be backported when the
> bug is critical enough to warrant the risk.
> - Users of supported versions benefit greatly from the resolution of
> bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
> for a given bug fix, it should be backported."
>
>
> Cheers,
> Greg
>
>
> On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
>> Back porting as little as possible is the ultimate goal for me. My
>> reasons are closely aligned with what Andrew wrote above.
>>
>> If we agree on this strategy, the next question is how to enforce it. My
>> intuition is that committers will lean towards back porting their patches
>> in arguable cases, because humans tend to overestimate the importance of
>> their personal work. Delegating the decision in such cases to a release
>> manager in my opinion will help us enforce the strategy of minimal number
>> backports. As a bonus, the release manager will have a much better
>> understanding of what's going on with the release, keyword: "more
>> ownership".
>>
>> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> andrew@schwartzmeyer.com> wrote:
>>
>>> I believe I fall somewhere between Alex and Ben.
>>>
>>> As for deciding what to backport or not, I lean toward Alex's view of
>>> backporting as little as possible (and agree with his criteria). My
>>> reasoning is that all changes can have unforeseen consequences, which I
>>> believe is something to be actively avoided in already released versions.
>>> The reason for backporting patches to fix regressions is the same as the
>>> reason to avoid backporting as much as possible: keep behavior consistent
>>> (and safe) within a release. With that as the goal of a branch in
>>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>>> fix CVEs and other critical/blocking issues.
>>>
>>> As for who should decide what to backport, I lean toward Ben's view of
>>> the burden being on the committer. I don't think we should add more work
>>> for release managers, and I think the committer/shepherd obviously has the
>>> most understanding of the context around changes proposed for backport.
>>>
>>> Here's an example of a recent bugfix which I backported:
>>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>>
>>> While normally I believe this change falls under "avoid due to
>>> unforeseen consequences," I made an exception as the bug was old, circa
>>> 2015, (indicating it had been an issue for others), and was causing
>>> recurring failures in testing. The fix itself was very small, meaning it
>>> was easier to evaluate for possible side effects, so I felt a little safer
>>> in that regard. The effect of not having the fix was a fatal and undesired
>>> crash, which furthermore left troublesome side effects on the system (you
>>> couldn't bring the agent back up). And lastly, a dependent project (DC/OS)
>>> wanted it in their next bump, which necessitated backporting to the release
>>> they were pulling in.
>>>
>>> I think in general we should backport only as necessary, and leave it on
>>> the committers to decide if backporting a particular change is necessary.
>>>
>>>
>>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>>
>>>> This is exactly where our views differ, Ben : )
>>>>
>>>> Ideally, I would like a release manager to have more ownership and less
>>>> manual work. In my imagination, a release manager has more power and
>>>> control about dates, features, backports and everything that is related
>>>> to
>>>> "their" branch. I would also like us to back port as little as
>>>> possible, to
>>>> simplify testing and releasing patch versions.
>>>>
>>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>>>> wrote:
>>>>
>>>> +user, I probably it would be good to hear from users as well.
>>>>>
>>>>> Please see the original proposal as well as Alex's proposal and let us
>>>>> know
>>>>> your thoughts.
>>>>>
>>>>> To continue the discussion from where Alex left off:
>>>>>
>>>>> > Other bugs and significant improvements, e.g., performance, may be
>>>>> back
>>>>> ported,
>>>>> the release manager should ideally be the one who decides on this.
>>>>>
>>>>> I'm a little puzzled by this, why is the release manager involved? As
>>>>> we
>>>>> already document, backports occur when the bug is fixed, so this
>>>>> happens in
>>>>> the steady state of development, not at release time. The release
>>>>> manager
>>>>> only comes in at the time of the release itself, at which point all
>>>>> backports have already happened and the release manager handles the
>>>>> release
>>>>> process. Only blocker level issues can stop the release and while the
>>>>> release manager has a strong say, we should generally agree on what
>>>>> consists of a release blocking issue.
>>>>>
>>>>> Just to clarify my workflow, I generally backport every bug fix I
>>>>> commit
>>>>> that applies cleanly, right after I commit it to master (with the
>>>>> exceptions I listed below).
>>>>>
>>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>>>> wrote:
>>>>>
>>>>> > I would like to back port as little as possible. I suggest the
>>>>> following
>>>>> > criteria:
>>>>> >
>>>>> > * By default, regressions are back ported to existing release
>>>>> branches. A
>>>>> > bug is considered a regression if the functionality is present in the
>>>>> > previous minor or patch version and is not affected by the bug there.
>>>>> >
>>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>>>> >
>>>>> > * Other bugs and significant improvements, e.g., performance, may be
>>>>> back
>>>>> > ported, the release manager should ideally be the one who decides on
>>>>> this.
>>>>> >
>>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > > Ben, thanks for the clarification. I'm in agreement with the
>>>>> points you
>>>>> > > made.
>>>>> > >
>>>>> > > Once we have consensus, would you mind updating the doc?
>>>>> > >
>>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <
>>>>> bmahler@apache.org>
>>>>> > > wrote:
>>>>> > >
>>>>> > > > I realized recently that we aren't all on the same page with
>>>>> > backporting.
>>>>> > > > We currently only document the following:
>>>>> > > >
>>>>> > > > "Typically the fix for an issue that is affecting supported
>>>>> releases
>>>>> > > lands
>>>>> > > > on the master branch and is then backported to the release
>>>>> branch(es).
>>>>> > In
>>>>> > > > rare cases, the fix might directly go into a release branch
>>>>> without
>>>>> > > landing
>>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>>>> > > >
>>>>> > > > This leaves room for interpretation about what lies outside of
>>>>> > "typical".
>>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
>>>>> like
>>>>> to
>>>>> > > hear
>>>>> > > > what others have in mind:
>>>>> > > >
>>>>> > > > * By default, bug fixes at any level should be backported to
>>>>> existing
>>>>> > > > release branches if it affects those releases. Especially
>>>>> important:
>>>>> > > > crashes, bugs in non-experimental features.
>>>>> > > >
>>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>>>> backport
>>>>> > > fixes
>>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>>>> > experimental
>>>>> > > > features.
>>>>> > > >
>>>>> > > > * Exceptional non-bug cases that can be backported: performance
>>>>> > > > improvements.
>>>>> > > >
>>>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>>>> which
>>>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>>>> that
>>>>> > > gives
>>>>> > > > everyone the right mindset for common cases and then discuss
>>>>> corner
>>>>> > cases
>>>>> > > > on-demand in the future.
>>>>> > > >
>>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>>
>>>
>>
>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
My impression is that we have two opposing schools of thought here:
1. Backport as little as possible, to avoid unforeseen consequences
2. Backport as much as proves practical, to eliminate bugs in supported
versions
Do other people agree with this assessment?
If so, how can we find common ground? One possible solution would be to
leave the decision on backporting up to the committer, without specifying a
project-wide policy. This seems to be the status quo, and would lead to
some variation across committers regarding what types of fixes are
backported. We could also choose to delegate the decision to the release
manager; I favor leaving the decision with the committer, to eliminate the
burden on release managers.
Here's a thought: rather than defining a prescriptive "policy" that we
expect committers to abide by, we could enumerate in the documentation the
competing concerns that we expect committers to consider when making
decisions on backports. The committing docs could read something like:
"When bug fixes are committed to master, the committer should evaluate the
fix to determine whether or not it should be backported to supported
versions. This is left to the committer, but they are expected to weigh the
following concerns when making the decision:
- Every backported change comes with a risk of unintended consequences.
The change should be carefully evaluated to ensure that such side-effects
are highly unlikely.
- As the complexity of applying a backport increases due to merge
conflicts, the likelihood of unintended consequences also increases. Bug
fixes which require extensive rebasing should only be backported when the
bug is critical enough to warrant the risk.
- Users of supported versions benefit greatly from the resolution of
bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
for a given bug fix, it should be backported."
Cheers,
Greg
On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Back porting as little as possible is the ultimate goal for me. My reasons
> are closely aligned with what Andrew wrote above.
>
> If we agree on this strategy, the next question is how to enforce it. My
> intuition is that committers will lean towards back porting their patches
> in arguable cases, because humans tend to overestimate the importance of
> their personal work. Delegating the decision in such cases to a release
> manager in my opinion will help us enforce the strategy of minimal number
> backports. As a bonus, the release manager will have a much better
> understanding of what's going on with the release, keyword: "more
> ownership".
>
> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> andrew@schwartzmeyer.com> wrote:
>
>> I believe I fall somewhere between Alex and Ben.
>>
>> As for deciding what to backport or not, I lean toward Alex's view of
>> backporting as little as possible (and agree with his criteria). My
>> reasoning is that all changes can have unforeseen consequences, which I
>> believe is something to be actively avoided in already released versions.
>> The reason for backporting patches to fix regressions is the same as the
>> reason to avoid backporting as much as possible: keep behavior consistent
>> (and safe) within a release. With that as the goal of a branch in
>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>> fix CVEs and other critical/blocking issues.
>>
>> As for who should decide what to backport, I lean toward Ben's view of
>> the burden being on the committer. I don't think we should add more work
>> for release managers, and I think the committer/shepherd obviously has the
>> most understanding of the context around changes proposed for backport.
>>
>> Here's an example of a recent bugfix which I backported:
>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>
>> While normally I believe this change falls under "avoid due to unforeseen
>> consequences," I made an exception as the bug was old, circa 2015,
>> (indicating it had been an issue for others), and was causing recurring
>> failures in testing. The fix itself was very small, meaning it was easier
>> to evaluate for possible side effects, so I felt a little safer in that
>> regard. The effect of not having the fix was a fatal and undesired crash,
>> which furthermore left troublesome side effects on the system (you couldn't
>> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
>> in their next bump, which necessitated backporting to the release they were
>> pulling in.
>>
>> I think in general we should backport only as necessary, and leave it on
>> the committers to decide if backporting a particular change is necessary.
>>
>>
>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>
>>> This is exactly where our views differ, Ben : )
>>>
>>> Ideally, I would like a release manager to have more ownership and less
>>> manual work. In my imagination, a release manager has more power and
>>> control about dates, features, backports and everything that is related
>>> to
>>> "their" branch. I would also like us to back port as little as possible,
>>> to
>>> simplify testing and releasing patch versions.
>>>
>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>>> wrote:
>>>
>>> +user, I probably it would be good to hear from users as well.
>>>>
>>>> Please see the original proposal as well as Alex's proposal and let us
>>>> know
>>>> your thoughts.
>>>>
>>>> To continue the discussion from where Alex left off:
>>>>
>>>> > Other bugs and significant improvements, e.g., performance, may be
>>>> back
>>>> ported,
>>>> the release manager should ideally be the one who decides on this.
>>>>
>>>> I'm a little puzzled by this, why is the release manager involved? As we
>>>> already document, backports occur when the bug is fixed, so this
>>>> happens in
>>>> the steady state of development, not at release time. The release
>>>> manager
>>>> only comes in at the time of the release itself, at which point all
>>>> backports have already happened and the release manager handles the
>>>> release
>>>> process. Only blocker level issues can stop the release and while the
>>>> release manager has a strong say, we should generally agree on what
>>>> consists of a release blocking issue.
>>>>
>>>> Just to clarify my workflow, I generally backport every bug fix I commit
>>>> that applies cleanly, right after I commit it to master (with the
>>>> exceptions I listed below).
>>>>
>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>>> wrote:
>>>>
>>>> > I would like to back port as little as possible. I suggest the
>>>> following
>>>> > criteria:
>>>> >
>>>> > * By default, regressions are back ported to existing release
>>>> branches. A
>>>> > bug is considered a regression if the functionality is present in the
>>>> > previous minor or patch version and is not affected by the bug there.
>>>> >
>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>>> >
>>>> > * Other bugs and significant improvements, e.g., performance, may be
>>>> back
>>>> > ported, the release manager should ideally be the one who decides on
>>>> this.
>>>> >
>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>>> wrote:
>>>> >
>>>> > > Ben, thanks for the clarification. I'm in agreement with the points
>>>> you
>>>> > > made.
>>>> > >
>>>> > > Once we have consensus, would you mind updating the doc?
>>>> > >
>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bmahler@apache.org
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > > I realized recently that we aren't all on the same page with
>>>> > backporting.
>>>> > > > We currently only document the following:
>>>> > > >
>>>> > > > "Typically the fix for an issue that is affecting supported
>>>> releases
>>>> > > lands
>>>> > > > on the master branch and is then backported to the release
>>>> branch(es).
>>>> > In
>>>> > > > rare cases, the fix might directly go into a release branch
>>>> without
>>>> > > landing
>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>>> > > >
>>>> > > > This leaves room for interpretation about what lies outside of
>>>> > "typical".
>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
>>>> like
>>>> to
>>>> > > hear
>>>> > > > what others have in mind:
>>>> > > >
>>>> > > > * By default, bug fixes at any level should be backported to
>>>> existing
>>>> > > > release branches if it affects those releases. Especially
>>>> important:
>>>> > > > crashes, bugs in non-experimental features.
>>>> > > >
>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>>> backport
>>>> > > fixes
>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>>> > experimental
>>>> > > > features.
>>>> > > >
>>>> > > > * Exceptional non-bug cases that can be backported: performance
>>>> > > > improvements.
>>>> > > >
>>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>>> which
>>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>>> that
>>>> > > gives
>>>> > > > everyone the right mindset for common cases and then discuss
>>>> corner
>>>> > cases
>>>> > > > on-demand in the future.
>>>> > > >
>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>>
>>
>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
My impression is that we have two opposing schools of thought here:
1. Backport as little as possible, to avoid unforeseen consequences
2. Backport as much as proves practical, to eliminate bugs in supported
versions
Do other people agree with this assessment?
If so, how can we find common ground? One possible solution would be to
leave the decision on backporting up to the committer, without specifying a
project-wide policy. This seems to be the status quo, and would lead to
some variation across committers regarding what types of fixes are
backported. We could also choose to delegate the decision to the release
manager; I favor leaving the decision with the committer, to eliminate the
burden on release managers.
Here's a thought: rather than defining a prescriptive "policy" that we
expect committers to abide by, we could enumerate in the documentation the
competing concerns that we expect committers to consider when making
decisions on backports. The committing docs could read something like:
"When bug fixes are committed to master, the committer should evaluate the
fix to determine whether or not it should be backported to supported
versions. This is left to the committer, but they are expected to weigh the
following concerns when making the decision:
- Every backported change comes with a risk of unintended consequences.
The change should be carefully evaluated to ensure that such side-effects
are highly unlikely.
- As the complexity of applying a backport increases due to merge
conflicts, the likelihood of unintended consequences also increases. Bug
fixes which require extensive rebasing should only be backported when the
bug is critical enough to warrant the risk.
- Users of supported versions benefit greatly from the resolution of
bugs in point releases. Thus, whenever concerns #1 and #2 can be allayed
for a given bug fix, it should be backported."
Cheers,
Greg
On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Back porting as little as possible is the ultimate goal for me. My reasons
> are closely aligned with what Andrew wrote above.
>
> If we agree on this strategy, the next question is how to enforce it. My
> intuition is that committers will lean towards back porting their patches
> in arguable cases, because humans tend to overestimate the importance of
> their personal work. Delegating the decision in such cases to a release
> manager in my opinion will help us enforce the strategy of minimal number
> backports. As a bonus, the release manager will have a much better
> understanding of what's going on with the release, keyword: "more
> ownership".
>
> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
> andrew@schwartzmeyer.com> wrote:
>
>> I believe I fall somewhere between Alex and Ben.
>>
>> As for deciding what to backport or not, I lean toward Alex's view of
>> backporting as little as possible (and agree with his criteria). My
>> reasoning is that all changes can have unforeseen consequences, which I
>> believe is something to be actively avoided in already released versions.
>> The reason for backporting patches to fix regressions is the same as the
>> reason to avoid backporting as much as possible: keep behavior consistent
>> (and safe) within a release. With that as the goal of a branch in
>> maintenance mode, it makes sense to fix regressions, and make exceptions to
>> fix CVEs and other critical/blocking issues.
>>
>> As for who should decide what to backport, I lean toward Ben's view of
>> the burden being on the committer. I don't think we should add more work
>> for release managers, and I think the committer/shepherd obviously has the
>> most understanding of the context around changes proposed for backport.
>>
>> Here's an example of a recent bugfix which I backported:
>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>
>> While normally I believe this change falls under "avoid due to unforeseen
>> consequences," I made an exception as the bug was old, circa 2015,
>> (indicating it had been an issue for others), and was causing recurring
>> failures in testing. The fix itself was very small, meaning it was easier
>> to evaluate for possible side effects, so I felt a little safer in that
>> regard. The effect of not having the fix was a fatal and undesired crash,
>> which furthermore left troublesome side effects on the system (you couldn't
>> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
>> in their next bump, which necessitated backporting to the release they were
>> pulling in.
>>
>> I think in general we should backport only as necessary, and leave it on
>> the committers to decide if backporting a particular change is necessary.
>>
>>
>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>
>>> This is exactly where our views differ, Ben : )
>>>
>>> Ideally, I would like a release manager to have more ownership and less
>>> manual work. In my imagination, a release manager has more power and
>>> control about dates, features, backports and everything that is related
>>> to
>>> "their" branch. I would also like us to back port as little as possible,
>>> to
>>> simplify testing and releasing patch versions.
>>>
>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>>> wrote:
>>>
>>> +user, I probably it would be good to hear from users as well.
>>>>
>>>> Please see the original proposal as well as Alex's proposal and let us
>>>> know
>>>> your thoughts.
>>>>
>>>> To continue the discussion from where Alex left off:
>>>>
>>>> > Other bugs and significant improvements, e.g., performance, may be
>>>> back
>>>> ported,
>>>> the release manager should ideally be the one who decides on this.
>>>>
>>>> I'm a little puzzled by this, why is the release manager involved? As we
>>>> already document, backports occur when the bug is fixed, so this
>>>> happens in
>>>> the steady state of development, not at release time. The release
>>>> manager
>>>> only comes in at the time of the release itself, at which point all
>>>> backports have already happened and the release manager handles the
>>>> release
>>>> process. Only blocker level issues can stop the release and while the
>>>> release manager has a strong say, we should generally agree on what
>>>> consists of a release blocking issue.
>>>>
>>>> Just to clarify my workflow, I generally backport every bug fix I commit
>>>> that applies cleanly, right after I commit it to master (with the
>>>> exceptions I listed below).
>>>>
>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>>> wrote:
>>>>
>>>> > I would like to back port as little as possible. I suggest the
>>>> following
>>>> > criteria:
>>>> >
>>>> > * By default, regressions are back ported to existing release
>>>> branches. A
>>>> > bug is considered a regression if the functionality is present in the
>>>> > previous minor or patch version and is not affected by the bug there.
>>>> >
>>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>>> >
>>>> > * Other bugs and significant improvements, e.g., performance, may be
>>>> back
>>>> > ported, the release manager should ideally be the one who decides on
>>>> this.
>>>> >
>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>>> wrote:
>>>> >
>>>> > > Ben, thanks for the clarification. I'm in agreement with the points
>>>> you
>>>> > > made.
>>>> > >
>>>> > > Once we have consensus, would you mind updating the doc?
>>>> > >
>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bmahler@apache.org
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > > I realized recently that we aren't all on the same page with
>>>> > backporting.
>>>> > > > We currently only document the following:
>>>> > > >
>>>> > > > "Typically the fix for an issue that is affecting supported
>>>> releases
>>>> > > lands
>>>> > > > on the master branch and is then backported to the release
>>>> branch(es).
>>>> > In
>>>> > > > rare cases, the fix might directly go into a release branch
>>>> without
>>>> > > landing
>>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>>> > > >
>>>> > > > This leaves room for interpretation about what lies outside of
>>>> > "typical".
>>>> > > > Here's the simplest way I can explain what I stick to, and I'd
>>>> like
>>>> to
>>>> > > hear
>>>> > > > what others have in mind:
>>>> > > >
>>>> > > > * By default, bug fixes at any level should be backported to
>>>> existing
>>>> > > > release branches if it affects those releases. Especially
>>>> important:
>>>> > > > crashes, bugs in non-experimental features.
>>>> > > >
>>>> > > > * Exceptional cases that can omit backporting: difficult to
>>>> backport
>>>> > > fixes
>>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>>> > experimental
>>>> > > > features.
>>>> > > >
>>>> > > > * Exceptional non-bug cases that can be backported: performance
>>>> > > > improvements.
>>>> > > >
>>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>>> which
>>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>>> that
>>>> > > gives
>>>> > > > everyone the right mindset for common cases and then discuss
>>>> corner
>>>> > cases
>>>> > > > on-demand in the future.
>>>> > > >
>>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>>
>>
>
Re: Backport Policy
Posted by Alex Rukletsov <al...@mesosphere.com>.
Back porting as little as possible is the ultimate goal for me. My reasons
are closely aligned with what Andrew wrote above.
If we agree on this strategy, the next question is how to enforce it. My
intuition is that committers will lean towards back porting their patches
in arguable cases, because humans tend to overestimate the importance of
their personal work. Delegating the decision in such cases to a release
manager in my opinion will help us enforce the strategy of minimal number
backports. As a bonus, the release manager will have a much better
understanding of what's going on with the release, keyword: "more
ownership".
On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
andrew@schwartzmeyer.com> wrote:
> I believe I fall somewhere between Alex and Ben.
>
> As for deciding what to backport or not, I lean toward Alex's view of
> backporting as little as possible (and agree with his criteria). My
> reasoning is that all changes can have unforeseen consequences, which I
> believe is something to be actively avoided in already released versions.
> The reason for backporting patches to fix regressions is the same as the
> reason to avoid backporting as much as possible: keep behavior consistent
> (and safe) within a release. With that as the goal of a branch in
> maintenance mode, it makes sense to fix regressions, and make exceptions to
> fix CVEs and other critical/blocking issues.
>
> As for who should decide what to backport, I lean toward Ben's view of the
> burden being on the committer. I don't think we should add more work for
> release managers, and I think the committer/shepherd obviously has the most
> understanding of the context around changes proposed for backport.
>
> Here's an example of a recent bugfix which I backported:
> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>
> While normally I believe this change falls under "avoid due to unforeseen
> consequences," I made an exception as the bug was old, circa 2015,
> (indicating it had been an issue for others), and was causing recurring
> failures in testing. The fix itself was very small, meaning it was easier
> to evaluate for possible side effects, so I felt a little safer in that
> regard. The effect of not having the fix was a fatal and undesired crash,
> which furthermore left troublesome side effects on the system (you couldn't
> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
> in their next bump, which necessitated backporting to the release they were
> pulling in.
>
> I think in general we should backport only as necessary, and leave it on
> the committers to decide if backporting a particular change is necessary.
>
>
> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>> wrote:
>>
>> +user, I probably it would be good to hear from users as well.
>>>
>>> Please see the original proposal as well as Alex's proposal and let us
>>> know
>>> your thoughts.
>>>
>>> To continue the discussion from where Alex left off:
>>>
>>> > Other bugs and significant improvements, e.g., performance, may be back
>>> ported,
>>> the release manager should ideally be the one who decides on this.
>>>
>>> I'm a little puzzled by this, why is the release manager involved? As we
>>> already document, backports occur when the bug is fixed, so this happens
>>> in
>>> the steady state of development, not at release time. The release manager
>>> only comes in at the time of the release itself, at which point all
>>> backports have already happened and the release manager handles the
>>> release
>>> process. Only blocker level issues can stop the release and while the
>>> release manager has a strong say, we should generally agree on what
>>> consists of a release blocking issue.
>>>
>>> Just to clarify my workflow, I generally backport every bug fix I commit
>>> that applies cleanly, right after I commit it to master (with the
>>> exceptions I listed below).
>>>
>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>> wrote:
>>>
>>> > I would like to back port as little as possible. I suggest the
>>> following
>>> > criteria:
>>> >
>>> > * By default, regressions are back ported to existing release
>>> branches. A
>>> > bug is considered a regression if the functionality is present in the
>>> > previous minor or patch version and is not affected by the bug there.
>>> >
>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>> >
>>> > * Other bugs and significant improvements, e.g., performance, may be
>>> back
>>> > ported, the release manager should ideally be the one who decides on
>>> this.
>>> >
>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>> wrote:
>>> >
>>> > > Ben, thanks for the clarification. I'm in agreement with the points
>>> you
>>> > > made.
>>> > >
>>> > > Once we have consensus, would you mind updating the doc?
>>> > >
>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
>>> > > wrote:
>>> > >
>>> > > > I realized recently that we aren't all on the same page with
>>> > backporting.
>>> > > > We currently only document the following:
>>> > > >
>>> > > > "Typically the fix for an issue that is affecting supported
>>> releases
>>> > > lands
>>> > > > on the master branch and is then backported to the release
>>> branch(es).
>>> > In
>>> > > > rare cases, the fix might directly go into a release branch without
>>> > > landing
>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>> > > >
>>> > > > This leaves room for interpretation about what lies outside of
>>> > "typical".
>>> > > > Here's the simplest way I can explain what I stick to, and I'd like
>>> to
>>> > > hear
>>> > > > what others have in mind:
>>> > > >
>>> > > > * By default, bug fixes at any level should be backported to
>>> existing
>>> > > > release branches if it affects those releases. Especially
>>> important:
>>> > > > crashes, bugs in non-experimental features.
>>> > > >
>>> > > > * Exceptional cases that can omit backporting: difficult to
>>> backport
>>> > > fixes
>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>> > experimental
>>> > > > features.
>>> > > >
>>> > > > * Exceptional non-bug cases that can be backported: performance
>>> > > > improvements.
>>> > > >
>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>> which
>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>> that
>>> > > gives
>>> > > > everyone the right mindset for common cases and then discuss corner
>>> > cases
>>> > > > on-demand in the future.
>>> > > >
>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>> > > >
>>> > >
>>> >
>>>
>>>
>
Re: Backport Policy
Posted by Alex Rukletsov <al...@mesosphere.com>.
Back porting as little as possible is the ultimate goal for me. My reasons
are closely aligned with what Andrew wrote above.
If we agree on this strategy, the next question is how to enforce it. My
intuition is that committers will lean towards back porting their patches
in arguable cases, because humans tend to overestimate the importance of
their personal work. Delegating the decision in such cases to a release
manager in my opinion will help us enforce the strategy of minimal number
backports. As a bonus, the release manager will have a much better
understanding of what's going on with the release, keyword: "more
ownership".
On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
andrew@schwartzmeyer.com> wrote:
> I believe I fall somewhere between Alex and Ben.
>
> As for deciding what to backport or not, I lean toward Alex's view of
> backporting as little as possible (and agree with his criteria). My
> reasoning is that all changes can have unforeseen consequences, which I
> believe is something to be actively avoided in already released versions.
> The reason for backporting patches to fix regressions is the same as the
> reason to avoid backporting as much as possible: keep behavior consistent
> (and safe) within a release. With that as the goal of a branch in
> maintenance mode, it makes sense to fix regressions, and make exceptions to
> fix CVEs and other critical/blocking issues.
>
> As for who should decide what to backport, I lean toward Ben's view of the
> burden being on the committer. I don't think we should add more work for
> release managers, and I think the committer/shepherd obviously has the most
> understanding of the context around changes proposed for backport.
>
> Here's an example of a recent bugfix which I backported:
> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>
> While normally I believe this change falls under "avoid due to unforeseen
> consequences," I made an exception as the bug was old, circa 2015,
> (indicating it had been an issue for others), and was causing recurring
> failures in testing. The fix itself was very small, meaning it was easier
> to evaluate for possible side effects, so I felt a little safer in that
> regard. The effect of not having the fix was a fatal and undesired crash,
> which furthermore left troublesome side effects on the system (you couldn't
> bring the agent back up). And lastly, a dependent project (DC/OS) wanted it
> in their next bump, which necessitated backporting to the release they were
> pulling in.
>
> I think in general we should backport only as necessary, and leave it on
> the committers to decide if backporting a particular change is necessary.
>
>
> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>> wrote:
>>
>> +user, I probably it would be good to hear from users as well.
>>>
>>> Please see the original proposal as well as Alex's proposal and let us
>>> know
>>> your thoughts.
>>>
>>> To continue the discussion from where Alex left off:
>>>
>>> > Other bugs and significant improvements, e.g., performance, may be back
>>> ported,
>>> the release manager should ideally be the one who decides on this.
>>>
>>> I'm a little puzzled by this, why is the release manager involved? As we
>>> already document, backports occur when the bug is fixed, so this happens
>>> in
>>> the steady state of development, not at release time. The release manager
>>> only comes in at the time of the release itself, at which point all
>>> backports have already happened and the release manager handles the
>>> release
>>> process. Only blocker level issues can stop the release and while the
>>> release manager has a strong say, we should generally agree on what
>>> consists of a release blocking issue.
>>>
>>> Just to clarify my workflow, I generally backport every bug fix I commit
>>> that applies cleanly, right after I commit it to master (with the
>>> exceptions I listed below).
>>>
>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>>> wrote:
>>>
>>> > I would like to back port as little as possible. I suggest the
>>> following
>>> > criteria:
>>> >
>>> > * By default, regressions are back ported to existing release
>>> branches. A
>>> > bug is considered a regression if the functionality is present in the
>>> > previous minor or patch version and is not affected by the bug there.
>>> >
>>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>>> >
>>> > * Other bugs and significant improvements, e.g., performance, may be
>>> back
>>> > ported, the release manager should ideally be the one who decides on
>>> this.
>>> >
>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>>> wrote:
>>> >
>>> > > Ben, thanks for the clarification. I'm in agreement with the points
>>> you
>>> > > made.
>>> > >
>>> > > Once we have consensus, would you mind updating the doc?
>>> > >
>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
>>> > > wrote:
>>> > >
>>> > > > I realized recently that we aren't all on the same page with
>>> > backporting.
>>> > > > We currently only document the following:
>>> > > >
>>> > > > "Typically the fix for an issue that is affecting supported
>>> releases
>>> > > lands
>>> > > > on the master branch and is then backported to the release
>>> branch(es).
>>> > In
>>> > > > rare cases, the fix might directly go into a release branch without
>>> > > landing
>>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>>> > > >
>>> > > > This leaves room for interpretation about what lies outside of
>>> > "typical".
>>> > > > Here's the simplest way I can explain what I stick to, and I'd like
>>> to
>>> > > hear
>>> > > > what others have in mind:
>>> > > >
>>> > > > * By default, bug fixes at any level should be backported to
>>> existing
>>> > > > release branches if it affects those releases. Especially
>>> important:
>>> > > > crashes, bugs in non-experimental features.
>>> > > >
>>> > > > * Exceptional cases that can omit backporting: difficult to
>>> backport
>>> > > fixes
>>> > > > (especially if the bugs are deemed of low priority), bugs in
>>> > experimental
>>> > > > features.
>>> > > >
>>> > > > * Exceptional non-bug cases that can be backported: performance
>>> > > > improvements.
>>> > > >
>>> > > > I realize that there is a ton of subtlety here (even in terms of
>>> which
>>> > > > things are defined as bugs). But I hope we can lay down a policy
>>> that
>>> > > gives
>>> > > > everyone the right mindset for common cases and then discuss corner
>>> > cases
>>> > > > on-demand in the future.
>>> > > >
>>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>>> > > >
>>> > >
>>> >
>>>
>>>
>
Re: Backport Policy
Posted by Andrew Schwartzmeyer <an...@schwartzmeyer.com>.
I believe I fall somewhere between Alex and Ben.
As for deciding what to backport or not, I lean toward Alex's view of
backporting as little as possible (and agree with his criteria). My
reasoning is that all changes can have unforeseen consequences, which I
believe is something to be actively avoided in already released
versions. The reason for backporting patches to fix regressions is the
same as the reason to avoid backporting as much as possible: keep
behavior consistent (and safe) within a release. With that as the goal
of a branch in maintenance mode, it makes sense to fix regressions, and
make exceptions to fix CVEs and other critical/blocking issues.
As for who should decide what to backport, I lean toward Ben's view of
the burden being on the committer. I don't think we should add more work
for release managers, and I think the committer/shepherd obviously has
the most understanding of the context around changes proposed for
backport.
Here's an example of a recent bugfix which I backported:
https://reviews.apache.org/r/67587/ (for MESOS-3790)
While normally I believe this change falls under "avoid due to
unforeseen consequences," I made an exception as the bug was old, circa
2015, (indicating it had been an issue for others), and was causing
recurring failures in testing. The fix itself was very small, meaning it
was easier to evaluate for possible side effects, so I felt a little
safer in that regard. The effect of not having the fix was a fatal and
undesired crash, which furthermore left troublesome side effects on the
system (you couldn't bring the agent back up). And lastly, a dependent
project (DC/OS) wanted it in their next bump, which necessitated
backporting to the release they were pulling in.
I think in general we should backport only as necessary, and leave it on
the committers to decide if backporting a particular change is
necessary.
On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> This is exactly where our views differ, Ben : )
>
> Ideally, I would like a release manager to have more ownership and less
> manual work. In my imagination, a release manager has more power and
> control about dates, features, backports and everything that is related
> to
> "their" branch. I would also like us to back port as little as
> possible, to
> simplify testing and releasing patch versions.
>
> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
>> +user, I probably it would be good to hear from users as well.
>>
>> Please see the original proposal as well as Alex's proposal and let us
>> know
>> your thoughts.
>>
>> To continue the discussion from where Alex left off:
>>
>> > Other bugs and significant improvements, e.g., performance, may be back
>> ported,
>> the release manager should ideally be the one who decides on this.
>>
>> I'm a little puzzled by this, why is the release manager involved? As
>> we
>> already document, backports occur when the bug is fixed, so this
>> happens in
>> the steady state of development, not at release time. The release
>> manager
>> only comes in at the time of the release itself, at which point all
>> backports have already happened and the release manager handles the
>> release
>> process. Only blocker level issues can stop the release and while the
>> release manager has a strong say, we should generally agree on what
>> consists of a release blocking issue.
>>
>> Just to clarify my workflow, I generally backport every bug fix I
>> commit
>> that applies cleanly, right after I commit it to master (with the
>> exceptions I listed below).
>>
>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>> wrote:
>>
>> > I would like to back port as little as possible. I suggest the following
>> > criteria:
>> >
>> > * By default, regressions are back ported to existing release branches. A
>> > bug is considered a regression if the functionality is present in the
>> > previous minor or patch version and is not affected by the bug there.
>> >
>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> >
>> > * Other bugs and significant improvements, e.g., performance, may be back
>> > ported, the release manager should ideally be the one who decides on
>> this.
>> >
>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>> wrote:
>> >
>> > > Ben, thanks for the clarification. I'm in agreement with the points you
>> > > made.
>> > >
>> > > Once we have consensus, would you mind updating the doc?
>> > >
>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
>> > > wrote:
>> > >
>> > > > I realized recently that we aren't all on the same page with
>> > backporting.
>> > > > We currently only document the following:
>> > > >
>> > > > "Typically the fix for an issue that is affecting supported releases
>> > > lands
>> > > > on the master branch and is then backported to the release
>> branch(es).
>> > In
>> > > > rare cases, the fix might directly go into a release branch without
>> > > landing
>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>> > > >
>> > > > This leaves room for interpretation about what lies outside of
>> > "typical".
>> > > > Here's the simplest way I can explain what I stick to, and I'd like
>> to
>> > > hear
>> > > > what others have in mind:
>> > > >
>> > > > * By default, bug fixes at any level should be backported to existing
>> > > > release branches if it affects those releases. Especially important:
>> > > > crashes, bugs in non-experimental features.
>> > > >
>> > > > * Exceptional cases that can omit backporting: difficult to backport
>> > > fixes
>> > > > (especially if the bugs are deemed of low priority), bugs in
>> > experimental
>> > > > features.
>> > > >
>> > > > * Exceptional non-bug cases that can be backported: performance
>> > > > improvements.
>> > > >
>> > > > I realize that there is a ton of subtlety here (even in terms of
>> which
>> > > > things are defined as bugs). But I hope we can lay down a policy that
>> > > gives
>> > > > everyone the right mindset for common cases and then discuss corner
>> > cases
>> > > > on-demand in the future.
>> > > >
>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>> > > >
>> > >
>> >
>>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
It seems to me that putting the burden of deciding on backports on the
release manager would actually increase the amount of work required. Simply
cutting the release on a particular date is pretty quick - however,
examining tickets to determine whether or not a particular fix should be
backported seems like more effort?
The backport policy strikes me as a community decision, rather than an
individual decision. The community discusses (as we're doing now :) and
settles on a policy, which individual committers then execute. Putting each
release manager in charge of the backport policy for that particular
release would lead to inconsistency across releases. I'm not sure that this
is a terrible thing, but it's not a particularly good thing either.
I would propose that we establish a single backport policy that we all do
our best to execute, with an understanding that there will always be room
for exceptions in some situations.
I like the idea of backporting all bug fixes which apply relatively
cleanly. In addition, very critical bug fixes are worth backporting even
when extensive work is required to backport them.
Alex, could you elaborate on why you would like to backport as little as
possible? I would like to better understand your motivations there :)
Cheers,
Greg
On Fri, Jul 13, 2018 at 2:40 PM, Jie Yu <yu...@gmail.com> wrote:
> I typically backport all bug fixes that cleanly apply and the risk is low.
> It's a judgement call, but many of the time, you can easily tell the risk
> is low.
>
> I think my argument on why we want to do this is "why not". I want our
> software to have less bugs!
>
> Letting release manager decides which patch to backport or not does not
> scale. Some release managers might even become dormant after a while.
>
> - Jie
>
> On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>> wrote:
>>
>> > +user, I probably it would be good to hear from users as well.
>> >
>> > Please see the original proposal as well as Alex's proposal and let us
>> know
>> > your thoughts.
>> >
>> > To continue the discussion from where Alex left off:
>> >
>> > > Other bugs and significant improvements, e.g., performance, may be
>> back
>> > ported,
>> > the release manager should ideally be the one who decides on this.
>> >
>> > I'm a little puzzled by this, why is the release manager involved? As we
>> > already document, backports occur when the bug is fixed, so this
>> happens in
>> > the steady state of development, not at release time. The release
>> manager
>> > only comes in at the time of the release itself, at which point all
>> > backports have already happened and the release manager handles the
>> release
>> > process. Only blocker level issues can stop the release and while the
>> > release manager has a strong say, we should generally agree on what
>> > consists of a release blocking issue.
>> >
>> > Just to clarify my workflow, I generally backport every bug fix I commit
>> > that applies cleanly, right after I commit it to master (with the
>> > exceptions I listed below).
>> >
>> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>> > wrote:
>> >
>> > > I would like to back port as little as possible. I suggest the
>> following
>> > > criteria:
>> > >
>> > > * By default, regressions are back ported to existing release
>> branches. A
>> > > bug is considered a regression if the functionality is present in the
>> > > previous minor or patch version and is not affected by the bug there.
>> > >
>> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> > >
>> > > * Other bugs and significant improvements, e.g., performance, may be
>> back
>> > > ported, the release manager should ideally be the one who decides on
>> > this.
>> > >
>> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>> > wrote:
>> > >
>> > > > Ben, thanks for the clarification. I'm in agreement with the points
>> you
>> > > > made.
>> > > >
>> > > > Once we have consensus, would you mind updating the doc?
>> > > >
>> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bmahler@apache.org
>> >
>> > > > wrote:
>> > > >
>> > > > > I realized recently that we aren't all on the same page with
>> > > backporting.
>> > > > > We currently only document the following:
>> > > > >
>> > > > > "Typically the fix for an issue that is affecting supported
>> releases
>> > > > lands
>> > > > > on the master branch and is then backported to the release
>> > branch(es).
>> > > In
>> > > > > rare cases, the fix might directly go into a release branch
>> without
>> > > > landing
>> > > > > on master (e.g., fix / issue is not applicable to master)." [1]
>> > > > >
>> > > > > This leaves room for interpretation about what lies outside of
>> > > "typical".
>> > > > > Here's the simplest way I can explain what I stick to, and I'd
>> like
>> > to
>> > > > hear
>> > > > > what others have in mind:
>> > > > >
>> > > > > * By default, bug fixes at any level should be backported to
>> existing
>> > > > > release branches if it affects those releases. Especially
>> important:
>> > > > > crashes, bugs in non-experimental features.
>> > > > >
>> > > > > * Exceptional cases that can omit backporting: difficult to
>> backport
>> > > > fixes
>> > > > > (especially if the bugs are deemed of low priority), bugs in
>> > > experimental
>> > > > > features.
>> > > > >
>> > > > > * Exceptional non-bug cases that can be backported: performance
>> > > > > improvements.
>> > > > >
>> > > > > I realize that there is a ton of subtlety here (even in terms of
>> > which
>> > > > > things are defined as bugs). But I hope we can lay down a policy
>> that
>> > > > gives
>> > > > > everyone the right mindset for common cases and then discuss
>> corner
>> > > cases
>> > > > > on-demand in the future.
>> > > > >
>> > > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Backport Policy
Posted by Greg Mann <gr...@mesosphere.io>.
It seems to me that putting the burden of deciding on backports on the
release manager would actually increase the amount of work required. Simply
cutting the release on a particular date is pretty quick - however,
examining tickets to determine whether or not a particular fix should be
backported seems like more effort?
The backport policy strikes me as a community decision, rather than an
individual decision. The community discusses (as we're doing now :) and
settles on a policy, which individual committers then execute. Putting each
release manager in charge of the backport policy for that particular
release would lead to inconsistency across releases. I'm not sure that this
is a terrible thing, but it's not a particularly good thing either.
I would propose that we establish a single backport policy that we all do
our best to execute, with an understanding that there will always be room
for exceptions in some situations.
I like the idea of backporting all bug fixes which apply relatively
cleanly. In addition, very critical bug fixes are worth backporting even
when extensive work is required to backport them.
Alex, could you elaborate on why you would like to backport as little as
possible? I would like to better understand your motivations there :)
Cheers,
Greg
On Fri, Jul 13, 2018 at 2:40 PM, Jie Yu <yu...@gmail.com> wrote:
> I typically backport all bug fixes that cleanly apply and the risk is low.
> It's a judgement call, but many of the time, you can easily tell the risk
> is low.
>
> I think my argument on why we want to do this is "why not". I want our
> software to have less bugs!
>
> Letting release manager decides which patch to backport or not does not
> scale. Some release managers might even become dormant after a while.
>
> - Jie
>
> On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
>> This is exactly where our views differ, Ben : )
>>
>> Ideally, I would like a release manager to have more ownership and less
>> manual work. In my imagination, a release manager has more power and
>> control about dates, features, backports and everything that is related to
>> "their" branch. I would also like us to back port as little as possible,
>> to
>> simplify testing and releasing patch versions.
>>
>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
>> wrote:
>>
>> > +user, I probably it would be good to hear from users as well.
>> >
>> > Please see the original proposal as well as Alex's proposal and let us
>> know
>> > your thoughts.
>> >
>> > To continue the discussion from where Alex left off:
>> >
>> > > Other bugs and significant improvements, e.g., performance, may be
>> back
>> > ported,
>> > the release manager should ideally be the one who decides on this.
>> >
>> > I'm a little puzzled by this, why is the release manager involved? As we
>> > already document, backports occur when the bug is fixed, so this
>> happens in
>> > the steady state of development, not at release time. The release
>> manager
>> > only comes in at the time of the release itself, at which point all
>> > backports have already happened and the release manager handles the
>> release
>> > process. Only blocker level issues can stop the release and while the
>> > release manager has a strong say, we should generally agree on what
>> > consists of a release blocking issue.
>> >
>> > Just to clarify my workflow, I generally backport every bug fix I commit
>> > that applies cleanly, right after I commit it to master (with the
>> > exceptions I listed below).
>> >
>> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>> > wrote:
>> >
>> > > I would like to back port as little as possible. I suggest the
>> following
>> > > criteria:
>> > >
>> > > * By default, regressions are back ported to existing release
>> branches. A
>> > > bug is considered a regression if the functionality is present in the
>> > > previous minor or patch version and is not affected by the bug there.
>> > >
>> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> > >
>> > > * Other bugs and significant improvements, e.g., performance, may be
>> back
>> > > ported, the release manager should ideally be the one who decides on
>> > this.
>> > >
>> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>> > wrote:
>> > >
>> > > > Ben, thanks for the clarification. I'm in agreement with the points
>> you
>> > > > made.
>> > > >
>> > > > Once we have consensus, would you mind updating the doc?
>> > > >
>> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bmahler@apache.org
>> >
>> > > > wrote:
>> > > >
>> > > > > I realized recently that we aren't all on the same page with
>> > > backporting.
>> > > > > We currently only document the following:
>> > > > >
>> > > > > "Typically the fix for an issue that is affecting supported
>> releases
>> > > > lands
>> > > > > on the master branch and is then backported to the release
>> > branch(es).
>> > > In
>> > > > > rare cases, the fix might directly go into a release branch
>> without
>> > > > landing
>> > > > > on master (e.g., fix / issue is not applicable to master)." [1]
>> > > > >
>> > > > > This leaves room for interpretation about what lies outside of
>> > > "typical".
>> > > > > Here's the simplest way I can explain what I stick to, and I'd
>> like
>> > to
>> > > > hear
>> > > > > what others have in mind:
>> > > > >
>> > > > > * By default, bug fixes at any level should be backported to
>> existing
>> > > > > release branches if it affects those releases. Especially
>> important:
>> > > > > crashes, bugs in non-experimental features.
>> > > > >
>> > > > > * Exceptional cases that can omit backporting: difficult to
>> backport
>> > > > fixes
>> > > > > (especially if the bugs are deemed of low priority), bugs in
>> > > experimental
>> > > > > features.
>> > > > >
>> > > > > * Exceptional non-bug cases that can be backported: performance
>> > > > > improvements.
>> > > > >
>> > > > > I realize that there is a ton of subtlety here (even in terms of
>> > which
>> > > > > things are defined as bugs). But I hope we can lay down a policy
>> that
>> > > > gives
>> > > > > everyone the right mindset for common cases and then discuss
>> corner
>> > > cases
>> > > > > on-demand in the future.
>> > > > >
>> > > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Backport Policy
Posted by Jie Yu <yu...@gmail.com>.
I typically backport all bug fixes that cleanly apply and the risk is low.
It's a judgement call, but many of the time, you can easily tell the risk
is low.
I think my argument on why we want to do this is "why not". I want our
software to have less bugs!
Letting release manager decides which patch to backport or not does not
scale. Some release managers might even become dormant after a while.
- Jie
On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov <al...@mesosphere.com>
wrote:
> This is exactly where our views differ, Ben : )
>
> Ideally, I would like a release manager to have more ownership and less
> manual work. In my imagination, a release manager has more power and
> control about dates, features, backports and everything that is related to
> "their" branch. I would also like us to back port as little as possible, to
> simplify testing and releasing patch versions.
>
> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
> > +user, I probably it would be good to hear from users as well.
> >
> > Please see the original proposal as well as Alex's proposal and let us
> know
> > your thoughts.
> >
> > To continue the discussion from where Alex left off:
> >
> > > Other bugs and significant improvements, e.g., performance, may be back
> > ported,
> > the release manager should ideally be the one who decides on this.
> >
> > I'm a little puzzled by this, why is the release manager involved? As we
> > already document, backports occur when the bug is fixed, so this happens
> in
> > the steady state of development, not at release time. The release manager
> > only comes in at the time of the release itself, at which point all
> > backports have already happened and the release manager handles the
> release
> > process. Only blocker level issues can stop the release and while the
> > release manager has a strong say, we should generally agree on what
> > consists of a release blocking issue.
> >
> > Just to clarify my workflow, I generally backport every bug fix I commit
> > that applies cleanly, right after I commit it to master (with the
> > exceptions I listed below).
> >
> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
> > wrote:
> >
> > > I would like to back port as little as possible. I suggest the
> following
> > > criteria:
> > >
> > > * By default, regressions are back ported to existing release
> branches. A
> > > bug is considered a regression if the functionality is present in the
> > > previous minor or patch version and is not affected by the bug there.
> > >
> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >
> > > * Other bugs and significant improvements, e.g., performance, may be
> back
> > > ported, the release manager should ideally be the one who decides on
> > this.
> > >
> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
> > wrote:
> > >
> > > > Ben, thanks for the clarification. I'm in agreement with the points
> you
> > > > made.
> > > >
> > > > Once we have consensus, would you mind updating the doc?
> > > >
> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
> > > > wrote:
> > > >
> > > > > I realized recently that we aren't all on the same page with
> > > backporting.
> > > > > We currently only document the following:
> > > > >
> > > > > "Typically the fix for an issue that is affecting supported
> releases
> > > > lands
> > > > > on the master branch and is then backported to the release
> > branch(es).
> > > In
> > > > > rare cases, the fix might directly go into a release branch without
> > > > landing
> > > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > > >
> > > > > This leaves room for interpretation about what lies outside of
> > > "typical".
> > > > > Here's the simplest way I can explain what I stick to, and I'd like
> > to
> > > > hear
> > > > > what others have in mind:
> > > > >
> > > > > * By default, bug fixes at any level should be backported to
> existing
> > > > > release branches if it affects those releases. Especially
> important:
> > > > > crashes, bugs in non-experimental features.
> > > > >
> > > > > * Exceptional cases that can omit backporting: difficult to
> backport
> > > > fixes
> > > > > (especially if the bugs are deemed of low priority), bugs in
> > > experimental
> > > > > features.
> > > > >
> > > > > * Exceptional non-bug cases that can be backported: performance
> > > > > improvements.
> > > > >
> > > > > I realize that there is a ton of subtlety here (even in terms of
> > which
> > > > > things are defined as bugs). But I hope we can lay down a policy
> that
> > > > gives
> > > > > everyone the right mindset for common cases and then discuss corner
> > > cases
> > > > > on-demand in the future.
> > > > >
> > > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > > >
> > > >
> > >
> >
>
Re: Backport Policy
Posted by Jie Yu <yu...@gmail.com>.
I typically backport all bug fixes that cleanly apply and the risk is low.
It's a judgement call, but many of the time, you can easily tell the risk
is low.
I think my argument on why we want to do this is "why not". I want our
software to have less bugs!
Letting release manager decides which patch to backport or not does not
scale. Some release managers might even become dormant after a while.
- Jie
On Fri, Jul 13, 2018 at 12:54 AM, Alex Rukletsov <al...@mesosphere.com>
wrote:
> This is exactly where our views differ, Ben : )
>
> Ideally, I would like a release manager to have more ownership and less
> manual work. In my imagination, a release manager has more power and
> control about dates, features, backports and everything that is related to
> "their" branch. I would also like us to back port as little as possible, to
> simplify testing and releasing patch versions.
>
> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
> > +user, I probably it would be good to hear from users as well.
> >
> > Please see the original proposal as well as Alex's proposal and let us
> know
> > your thoughts.
> >
> > To continue the discussion from where Alex left off:
> >
> > > Other bugs and significant improvements, e.g., performance, may be back
> > ported,
> > the release manager should ideally be the one who decides on this.
> >
> > I'm a little puzzled by this, why is the release manager involved? As we
> > already document, backports occur when the bug is fixed, so this happens
> in
> > the steady state of development, not at release time. The release manager
> > only comes in at the time of the release itself, at which point all
> > backports have already happened and the release manager handles the
> release
> > process. Only blocker level issues can stop the release and while the
> > release manager has a strong say, we should generally agree on what
> > consists of a release blocking issue.
> >
> > Just to clarify my workflow, I generally backport every bug fix I commit
> > that applies cleanly, right after I commit it to master (with the
> > exceptions I listed below).
> >
> > On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
> > wrote:
> >
> > > I would like to back port as little as possible. I suggest the
> following
> > > criteria:
> > >
> > > * By default, regressions are back ported to existing release
> branches. A
> > > bug is considered a regression if the functionality is present in the
> > > previous minor or patch version and is not affected by the bug there.
> > >
> > > * Critical and blocker issues, e.g., a CVE, can be back ported.
> > >
> > > * Other bugs and significant improvements, e.g., performance, may be
> back
> > > ported, the release manager should ideally be the one who decides on
> > this.
> > >
> > > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
> > wrote:
> > >
> > > > Ben, thanks for the clarification. I'm in agreement with the points
> you
> > > > made.
> > > >
> > > > Once we have consensus, would you mind updating the doc?
> > > >
> > > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
> > > > wrote:
> > > >
> > > > > I realized recently that we aren't all on the same page with
> > > backporting.
> > > > > We currently only document the following:
> > > > >
> > > > > "Typically the fix for an issue that is affecting supported
> releases
> > > > lands
> > > > > on the master branch and is then backported to the release
> > branch(es).
> > > In
> > > > > rare cases, the fix might directly go into a release branch without
> > > > landing
> > > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > > >
> > > > > This leaves room for interpretation about what lies outside of
> > > "typical".
> > > > > Here's the simplest way I can explain what I stick to, and I'd like
> > to
> > > > hear
> > > > > what others have in mind:
> > > > >
> > > > > * By default, bug fixes at any level should be backported to
> existing
> > > > > release branches if it affects those releases. Especially
> important:
> > > > > crashes, bugs in non-experimental features.
> > > > >
> > > > > * Exceptional cases that can omit backporting: difficult to
> backport
> > > > fixes
> > > > > (especially if the bugs are deemed of low priority), bugs in
> > > experimental
> > > > > features.
> > > > >
> > > > > * Exceptional non-bug cases that can be backported: performance
> > > > > improvements.
> > > > >
> > > > > I realize that there is a ton of subtlety here (even in terms of
> > which
> > > > > things are defined as bugs). But I hope we can lay down a policy
> that
> > > > gives
> > > > > everyone the right mindset for common cases and then discuss corner
> > > cases
> > > > > on-demand in the future.
> > > > >
> > > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > > >
> > > >
> > >
> >
>
Re: Backport Policy
Posted by Andrew Schwartzmeyer <an...@schwartzmeyer.com>.
I believe I fall somewhere between Alex and Ben.
As for deciding what to backport or not, I lean toward Alex's view of
backporting as little as possible (and agree with his criteria). My
reasoning is that all changes can have unforeseen consequences, which I
believe is something to be actively avoided in already released
versions. The reason for backporting patches to fix regressions is the
same as the reason to avoid backporting as much as possible: keep
behavior consistent (and safe) within a release. With that as the goal
of a branch in maintenance mode, it makes sense to fix regressions, and
make exceptions to fix CVEs and other critical/blocking issues.
As for who should decide what to backport, I lean toward Ben's view of
the burden being on the committer. I don't think we should add more work
for release managers, and I think the committer/shepherd obviously has
the most understanding of the context around changes proposed for
backport.
Here's an example of a recent bugfix which I backported:
https://reviews.apache.org/r/67587/ (for MESOS-3790)
While normally I believe this change falls under "avoid due to
unforeseen consequences," I made an exception as the bug was old, circa
2015, (indicating it had been an issue for others), and was causing
recurring failures in testing. The fix itself was very small, meaning it
was easier to evaluate for possible side effects, so I felt a little
safer in that regard. The effect of not having the fix was a fatal and
undesired crash, which furthermore left troublesome side effects on the
system (you couldn't bring the agent back up). And lastly, a dependent
project (DC/OS) wanted it in their next bump, which necessitated
backporting to the release they were pulling in.
I think in general we should backport only as necessary, and leave it on
the committers to decide if backporting a particular change is
necessary.
On 07/13/2018 12:54 am, Alex Rukletsov wrote:
> This is exactly where our views differ, Ben : )
>
> Ideally, I would like a release manager to have more ownership and less
> manual work. In my imagination, a release manager has more power and
> control about dates, features, backports and everything that is related
> to
> "their" branch. I would also like us to back port as little as
> possible, to
> simplify testing and releasing patch versions.
>
> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
>> +user, I probably it would be good to hear from users as well.
>>
>> Please see the original proposal as well as Alex's proposal and let us
>> know
>> your thoughts.
>>
>> To continue the discussion from where Alex left off:
>>
>> > Other bugs and significant improvements, e.g., performance, may be back
>> ported,
>> the release manager should ideally be the one who decides on this.
>>
>> I'm a little puzzled by this, why is the release manager involved? As
>> we
>> already document, backports occur when the bug is fixed, so this
>> happens in
>> the steady state of development, not at release time. The release
>> manager
>> only comes in at the time of the release itself, at which point all
>> backports have already happened and the release manager handles the
>> release
>> process. Only blocker level issues can stop the release and while the
>> release manager has a strong say, we should generally agree on what
>> consists of a release blocking issue.
>>
>> Just to clarify my workflow, I generally backport every bug fix I
>> commit
>> that applies cleanly, right after I commit it to master (with the
>> exceptions I listed below).
>>
>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
>> wrote:
>>
>> > I would like to back port as little as possible. I suggest the following
>> > criteria:
>> >
>> > * By default, regressions are back ported to existing release branches. A
>> > bug is considered a regression if the functionality is present in the
>> > previous minor or patch version and is not affected by the bug there.
>> >
>> > * Critical and blocker issues, e.g., a CVE, can be back ported.
>> >
>> > * Other bugs and significant improvements, e.g., performance, may be back
>> > ported, the release manager should ideally be the one who decides on
>> this.
>> >
>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
>> wrote:
>> >
>> > > Ben, thanks for the clarification. I'm in agreement with the points you
>> > > made.
>> > >
>> > > Once we have consensus, would you mind updating the doc?
>> > >
>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
>> > > wrote:
>> > >
>> > > > I realized recently that we aren't all on the same page with
>> > backporting.
>> > > > We currently only document the following:
>> > > >
>> > > > "Typically the fix for an issue that is affecting supported releases
>> > > lands
>> > > > on the master branch and is then backported to the release
>> branch(es).
>> > In
>> > > > rare cases, the fix might directly go into a release branch without
>> > > landing
>> > > > on master (e.g., fix / issue is not applicable to master)." [1]
>> > > >
>> > > > This leaves room for interpretation about what lies outside of
>> > "typical".
>> > > > Here's the simplest way I can explain what I stick to, and I'd like
>> to
>> > > hear
>> > > > what others have in mind:
>> > > >
>> > > > * By default, bug fixes at any level should be backported to existing
>> > > > release branches if it affects those releases. Especially important:
>> > > > crashes, bugs in non-experimental features.
>> > > >
>> > > > * Exceptional cases that can omit backporting: difficult to backport
>> > > fixes
>> > > > (especially if the bugs are deemed of low priority), bugs in
>> > experimental
>> > > > features.
>> > > >
>> > > > * Exceptional non-bug cases that can be backported: performance
>> > > > improvements.
>> > > >
>> > > > I realize that there is a ton of subtlety here (even in terms of
>> which
>> > > > things are defined as bugs). But I hope we can lay down a policy that
>> > > gives
>> > > > everyone the right mindset for common cases and then discuss corner
>> > cases
>> > > > on-demand in the future.
>> > > >
>> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
>> > > >
>> > >
>> >
>>
Re: Backport Policy
Posted by Alex Rukletsov <al...@mesosphere.com>.
This is exactly where our views differ, Ben : )
Ideally, I would like a release manager to have more ownership and less
manual work. In my imagination, a release manager has more power and
control about dates, features, backports and everything that is related to
"their" branch. I would also like us to back port as little as possible, to
simplify testing and releasing patch versions.
On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org> wrote:
> +user, I probably it would be good to hear from users as well.
>
> Please see the original proposal as well as Alex's proposal and let us know
> your thoughts.
>
> To continue the discussion from where Alex left off:
>
> > Other bugs and significant improvements, e.g., performance, may be back
> ported,
> the release manager should ideally be the one who decides on this.
>
> I'm a little puzzled by this, why is the release manager involved? As we
> already document, backports occur when the bug is fixed, so this happens in
> the steady state of development, not at release time. The release manager
> only comes in at the time of the release itself, at which point all
> backports have already happened and the release manager handles the release
> process. Only blocker level issues can stop the release and while the
> release manager has a strong say, we should generally agree on what
> consists of a release blocking issue.
>
> Just to clarify my workflow, I generally backport every bug fix I commit
> that applies cleanly, right after I commit it to master (with the
> exceptions I listed below).
>
> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
> > I would like to back port as little as possible. I suggest the following
> > criteria:
> >
> > * By default, regressions are back ported to existing release branches. A
> > bug is considered a regression if the functionality is present in the
> > previous minor or patch version and is not affected by the bug there.
> >
> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> >
> > * Other bugs and significant improvements, e.g., performance, may be back
> > ported, the release manager should ideally be the one who decides on
> this.
> >
> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
> wrote:
> >
> > > Ben, thanks for the clarification. I'm in agreement with the points you
> > > made.
> > >
> > > Once we have consensus, would you mind updating the doc?
> > >
> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
> > > wrote:
> > >
> > > > I realized recently that we aren't all on the same page with
> > backporting.
> > > > We currently only document the following:
> > > >
> > > > "Typically the fix for an issue that is affecting supported releases
> > > lands
> > > > on the master branch and is then backported to the release
> branch(es).
> > In
> > > > rare cases, the fix might directly go into a release branch without
> > > landing
> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > >
> > > > This leaves room for interpretation about what lies outside of
> > "typical".
> > > > Here's the simplest way I can explain what I stick to, and I'd like
> to
> > > hear
> > > > what others have in mind:
> > > >
> > > > * By default, bug fixes at any level should be backported to existing
> > > > release branches if it affects those releases. Especially important:
> > > > crashes, bugs in non-experimental features.
> > > >
> > > > * Exceptional cases that can omit backporting: difficult to backport
> > > fixes
> > > > (especially if the bugs are deemed of low priority), bugs in
> > experimental
> > > > features.
> > > >
> > > > * Exceptional non-bug cases that can be backported: performance
> > > > improvements.
> > > >
> > > > I realize that there is a ton of subtlety here (even in terms of
> which
> > > > things are defined as bugs). But I hope we can lay down a policy that
> > > gives
> > > > everyone the right mindset for common cases and then discuss corner
> > cases
> > > > on-demand in the future.
> > > >
> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > >
> > >
> >
>
Re: Backport Policy
Posted by Alex Rukletsov <al...@mesosphere.com>.
This is exactly where our views differ, Ben : )
Ideally, I would like a release manager to have more ownership and less
manual work. In my imagination, a release manager has more power and
control about dates, features, backports and everything that is related to
"their" branch. I would also like us to back port as little as possible, to
simplify testing and releasing patch versions.
On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <bm...@apache.org> wrote:
> +user, I probably it would be good to hear from users as well.
>
> Please see the original proposal as well as Alex's proposal and let us know
> your thoughts.
>
> To continue the discussion from where Alex left off:
>
> > Other bugs and significant improvements, e.g., performance, may be back
> ported,
> the release manager should ideally be the one who decides on this.
>
> I'm a little puzzled by this, why is the release manager involved? As we
> already document, backports occur when the bug is fixed, so this happens in
> the steady state of development, not at release time. The release manager
> only comes in at the time of the release itself, at which point all
> backports have already happened and the release manager handles the release
> process. Only blocker level issues can stop the release and while the
> release manager has a strong say, we should generally agree on what
> consists of a release blocking issue.
>
> Just to clarify my workflow, I generally backport every bug fix I commit
> that applies cleanly, right after I commit it to master (with the
> exceptions I listed below).
>
> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
>
> > I would like to back port as little as possible. I suggest the following
> > criteria:
> >
> > * By default, regressions are back ported to existing release branches. A
> > bug is considered a regression if the functionality is present in the
> > previous minor or patch version and is not affected by the bug there.
> >
> > * Critical and blocker issues, e.g., a CVE, can be back ported.
> >
> > * Other bugs and significant improvements, e.g., performance, may be back
> > ported, the release manager should ideally be the one who decides on
> this.
> >
> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <vi...@apache.org>
> wrote:
> >
> > > Ben, thanks for the clarification. I'm in agreement with the points you
> > > made.
> > >
> > > Once we have consensus, would you mind updating the doc?
> > >
> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler <bm...@apache.org>
> > > wrote:
> > >
> > > > I realized recently that we aren't all on the same page with
> > backporting.
> > > > We currently only document the following:
> > > >
> > > > "Typically the fix for an issue that is affecting supported releases
> > > lands
> > > > on the master branch and is then backported to the release
> branch(es).
> > In
> > > > rare cases, the fix might directly go into a release branch without
> > > landing
> > > > on master (e.g., fix / issue is not applicable to master)." [1]
> > > >
> > > > This leaves room for interpretation about what lies outside of
> > "typical".
> > > > Here's the simplest way I can explain what I stick to, and I'd like
> to
> > > hear
> > > > what others have in mind:
> > > >
> > > > * By default, bug fixes at any level should be backported to existing
> > > > release branches if it affects those releases. Especially important:
> > > > crashes, bugs in non-experimental features.
> > > >
> > > > * Exceptional cases that can omit backporting: difficult to backport
> > > fixes
> > > > (especially if the bugs are deemed of low priority), bugs in
> > experimental
> > > > features.
> > > >
> > > > * Exceptional non-bug cases that can be backported: performance
> > > > improvements.
> > > >
> > > > I realize that there is a ton of subtlety here (even in terms of
> which
> > > > things are defined as bugs). But I hope we can lay down a policy that
> > > gives
> > > > everyone the right mindset for common cases and then discuss corner
> > cases
> > > > on-demand in the future.
> > > >
> > > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > > >
> > >
> >
>