[DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Udo Kohlmeyer <ud...@apache.com>]

Hi there Apache Geode devs,

It has been some weeks since the proposed 1.10 release was cut. We've 
gone through a few cycles where we keep on submitting "please include 
ticket GEODE-XXX" because it is critical and will break the system. 
WHICH in reality tells me that current develop is broken and unstable.

I'm going to suggest that we abandon the current 1.10 release branch. I 
cannot shake the feeling that our continuous cherry picking into a 
branch will result in either the branch becoming unmaintainable, given 
we have only select fixes in the branch OR we end up with a branch that 
is more stable than our current development branch, which would result 
in us having to rebase the develop branch onto the 1.10 branch.

I propose that once we see the pipeline is clean and green for a solid 
we then again attempt to cut 1.10 branch.

We CANNOT continue adding to a branch in order to stabilize it.. It just 
means the branch we cut from is unstable. If we cannot cut a branch from 
develop without having to have weeks of stabilization cycles, then our 
main branch is broken...

Either way, not a good spot to be in.

Thoughts?

--Udo

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Owen Nichols <on...@pivotal.io>]

Udo, it sounds like you would like to modify the release process to start the stabilization work on develop (rather than cutting the branch as the first step).  I would love to hear community opinion on this proposal.

For 1.10, in the time we have cherry-picked 11 fixes to release/1.10.0, develop has seen over 90 commits.  I believe due process has been followed in bringing only critical fixes to stabilize the release branch.  Getting 1.10.0.RC1 into the hands of the community is the most important next step.  If there are no further critical fix proposals, Dick and I would like to proceed with the RC1 this week.

-Owen

> On Aug 26, 2019, at 4:06 PM, Udo Kohlmeyer <ud...@apache.com> wrote:
> 
> Hi there Apache Geode devs,
> 
> It has been some weeks since the proposed 1.10 release was cut. We've gone through a few cycles where we keep on submitting "please include ticket GEODE-XXX" because it is critical and will break the system. WHICH in reality tells me that current develop is broken and unstable.
> 
> I'm going to suggest that we abandon the current 1.10 release branch. I cannot shake the feeling that our continuous cherry picking into a branch will result in either the branch becoming unmaintainable, given we have only select fixes in the branch OR we end up with a branch that is more stable than our current development branch, which would result in us having to rebase the develop branch onto the 1.10 branch.
> 
> I propose that once we see the pipeline is clean and green for a solid we then again attempt to cut 1.10 branch.
> 
> We CANNOT continue adding to a branch in order to stabilize it.. It just means the branch we cut from is unstable. If we cannot cut a branch from develop without having to have weeks of stabilization cycles, then our main branch is broken...
> 
> Either way, not a good spot to be in.
> 
> Thoughts?
> 
> --Udo
> 

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Ryan McMahon <rm...@pivotal.io>]

+1 to continuing with the release branch we have.

I don't think that anybody is aiming to cause instability on the develop
branch.  We are all designing, writing code, refactoring, and writing
exhaustive tests to the best of our ability to ensure high quality.  We
have to accept that there will sometimes be unforeseen consequences; that
is the reality of working on a complex enterprise product.  I think that it
is unfortunate on this particular release we have seen as many critical
issues as we have, but I do not think there is anything fundamentally wrong
with our workflow.  I think if anything, cutting the release early and
allowing ample time to identify issues is a very good thing.  Yes, in
theory the develop branch should be releasable at any given moment, but in
reality there is sometimes a feedback delay where issues arise after some
time.  I think our current process of cutting a branch and cherry-picking
fixes for issues found allows us to deal with this delayed feedback.

The issues here to me are more around the delayed feedback time (issues
found by very rare race conditions that don't arise often) and a lack of
existing test coverage for many of our features.  I do believe that
everyone who has been committing refactored or new code is doing their due
diligence to write tests, but it is not always easy to identify where test
coverage gaps exist.  We will of course continue to fill those gaps as we
find them.  Do you have other ideas on how to make develop more stable?
Again, I personally feel people are doing their best to write new tests and
review code - to me this is not a process problem (more like a tech
debt/ease of refactoring problem) but I'm definitely open to discussion.

Ryan

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Udo Kohlmeyer <ud...@apache.com>]

Fundamentally I also don't have a problem with cherry-picking fixes to a 
potential release candidate. My concern is that the amount of fixes that 
were cherry-picked.

I also believe that "stabilizing" a release does have some 
cherry-picking, BUT the amount of stabilization that we have to apply 
has to be within reason and control.

I also understand that there are fixes that we want to include that are 
critical. BUT if it takes a month (from cutting to generating an RC) to 
get close to releasing, indicates that whatever is in develop is NOT 
stable. There were fixes that were included and the cause of the issues 
was "due to a recent commit SHA-- XXXX". Which means most likely the 
rigor to review and test said bug/issue was not effectively tested.

Finding issues with maps that should have been concurrent maps and lead 
memory leaks are distressing. Now there we have code that has been 
refactored that we prefer not to release. If we don't want to release 
it, why is it in the main develop branch. Imo, only code THAT CAN BE 
RELEASED should ever make it to the main branch.

Maybe I am the problem here (again).... Maybe it is I that is concerned 
with the quality that is being committed and then glossed over with, it 
is ok, we can fix any resulting issues coming from that.

If everybody is happy to release 1.10, then so be it... here is my +0.

But I don't think this approach of cutting a release branch and then 
effectively having to maintain to development branches and what commits 
are applied to the correct branches is in any way maintainable..

--Udo

On 8/27/19 11:56 AM, Bruce Schuchardt wrote:
> +1 for going ahead with the current release/1.10
>
> On 8/27/19 11:31 AM, Dan Smith wrote:
>> +1 to creating RC1 with the current release/1.10 branch this week.
>>
>> I don't see a fundamental problem with cherry-picking some targeted and
>> tested fixes to release/1.10, based on our assessment of the risk to
>> customers vs. the risk of destabilizing the branch. I think 
>> release/1.10 is
>> in a good state, and we should go ahead with the release.
>>
>> -Dan
>>
>>
>> On Tue, Aug 27, 2019 at 9:28 AM Bruce Schuchardt 
>> <bs...@pivotal.io>
>> wrote:
>>
>>> The "develop" branch has a refactoring of membership code that should
>>> not be included in 1.10.  I waited until the release branch was cut to
>>> push these changes.
>>>
>>> On 8/26/19 4:06 PM, Udo Kohlmeyer wrote:
>>>> Hi there Apache Geode devs,
>>>>
>>>> It has been some weeks since the proposed 1.10 release was cut. We've
>>>> gone through a few cycles where we keep on submitting "please include
>>>> ticket GEODE-XXX" because it is critical and will break the system.
>>>> WHICH in reality tells me that current develop is broken and unstable.
>>>>
>>>> I'm going to suggest that we abandon the current 1.10 release branch.
>>>> I cannot shake the feeling that our continuous cherry picking into a
>>>> branch will result in either the branch becoming unmaintainable, given
>>>> we have only select fixes in the branch OR we end up with a branch
>>>> that is more stable than our current development branch, which would
>>>> result in us having to rebase the develop branch onto the 1.10 branch.
>>>>
>>>> I propose that once we see the pipeline is clean and green for a solid
>>>> we then again attempt to cut 1.10 branch.
>>>>
>>>> We CANNOT continue adding to a branch in order to stabilize it.. It
>>>> just means the branch we cut from is unstable. If we cannot cut a
>>>> branch from develop without having to have weeks of stabilization
>>>> cycles, then our main branch is broken...
>>>>
>>>> Either way, not a good spot to be in.
>>>>
>>>> Thoughts?
>>>>
>>>> --Udo
>>>>

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Bruce Schuchardt <bs...@pivotal.io>]

+1 for going ahead with the current release/1.10

On 8/27/19 11:31 AM, Dan Smith wrote:
> +1 to creating RC1 with the current release/1.10 branch this week.
>
> I don't see a fundamental problem with cherry-picking some targeted and
> tested fixes to release/1.10, based on our assessment of the risk to
> customers vs. the risk of destabilizing the branch. I think release/1.10 is
> in a good state, and we should go ahead with the release.
>
> -Dan
>
>
> On Tue, Aug 27, 2019 at 9:28 AM Bruce Schuchardt <bs...@pivotal.io>
> wrote:
>
>> The "develop" branch has a refactoring of membership code that should
>> not be included in 1.10.  I waited until the release branch was cut to
>> push these changes.
>>
>> On 8/26/19 4:06 PM, Udo Kohlmeyer wrote:
>>> Hi there Apache Geode devs,
>>>
>>> It has been some weeks since the proposed 1.10 release was cut. We've
>>> gone through a few cycles where we keep on submitting "please include
>>> ticket GEODE-XXX" because it is critical and will break the system.
>>> WHICH in reality tells me that current develop is broken and unstable.
>>>
>>> I'm going to suggest that we abandon the current 1.10 release branch.
>>> I cannot shake the feeling that our continuous cherry picking into a
>>> branch will result in either the branch becoming unmaintainable, given
>>> we have only select fixes in the branch OR we end up with a branch
>>> that is more stable than our current development branch, which would
>>> result in us having to rebase the develop branch onto the 1.10 branch.
>>>
>>> I propose that once we see the pipeline is clean and green for a solid
>>> we then again attempt to cut 1.10 branch.
>>>
>>> We CANNOT continue adding to a branch in order to stabilize it.. It
>>> just means the branch we cut from is unstable. If we cannot cut a
>>> branch from develop without having to have weeks of stabilization
>>> cycles, then our main branch is broken...
>>>
>>> Either way, not a good spot to be in.
>>>
>>> Thoughts?
>>>
>>> --Udo
>>>

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Dan Smith <ds...@pivotal.io>]

+1 to creating RC1 with the current release/1.10 branch this week.

I don't see a fundamental problem with cherry-picking some targeted and
tested fixes to release/1.10, based on our assessment of the risk to
customers vs. the risk of destabilizing the branch. I think release/1.10 is
in a good state, and we should go ahead with the release.

-Dan


On Tue, Aug 27, 2019 at 9:28 AM Bruce Schuchardt <bs...@pivotal.io>
wrote:

> The "develop" branch has a refactoring of membership code that should
> not be included in 1.10.  I waited until the release branch was cut to
> push these changes.
>
> On 8/26/19 4:06 PM, Udo Kohlmeyer wrote:
> > Hi there Apache Geode devs,
> >
> > It has been some weeks since the proposed 1.10 release was cut. We've
> > gone through a few cycles where we keep on submitting "please include
> > ticket GEODE-XXX" because it is critical and will break the system.
> > WHICH in reality tells me that current develop is broken and unstable.
> >
> > I'm going to suggest that we abandon the current 1.10 release branch.
> > I cannot shake the feeling that our continuous cherry picking into a
> > branch will result in either the branch becoming unmaintainable, given
> > we have only select fixes in the branch OR we end up with a branch
> > that is more stable than our current development branch, which would
> > result in us having to rebase the develop branch onto the 1.10 branch.
> >
> > I propose that once we see the pipeline is clean and green for a solid
> > we then again attempt to cut 1.10 branch.
> >
> > We CANNOT continue adding to a branch in order to stabilize it.. It
> > just means the branch we cut from is unstable. If we cannot cut a
> > branch from develop without having to have weeks of stabilization
> > cycles, then our main branch is broken...
> >
> > Either way, not a good spot to be in.
> >
> > Thoughts?
> >
> > --Udo
> >
>

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Bruce Schuchardt <bs...@pivotal.io>]

The "develop" branch has a refactoring of membership code that should 
not be included in 1.10.  I waited until the release branch was cut to 
push these changes.

On 8/26/19 4:06 PM, Udo Kohlmeyer wrote:
> Hi there Apache Geode devs,
>
> It has been some weeks since the proposed 1.10 release was cut. We've 
> gone through a few cycles where we keep on submitting "please include 
> ticket GEODE-XXX" because it is critical and will break the system. 
> WHICH in reality tells me that current develop is broken and unstable.
>
> I'm going to suggest that we abandon the current 1.10 release branch. 
> I cannot shake the feeling that our continuous cherry picking into a 
> branch will result in either the branch becoming unmaintainable, given 
> we have only select fixes in the branch OR we end up with a branch 
> that is more stable than our current development branch, which would 
> result in us having to rebase the develop branch onto the 1.10 branch.
>
> I propose that once we see the pipeline is clean and green for a solid 
> we then again attempt to cut 1.10 branch.
>
> We CANNOT continue adding to a branch in order to stabilize it.. It 
> just means the branch we cut from is unstable. If we cannot cut a 
> branch from develop without having to have weeks of stabilization 
> cycles, then our main branch is broken...
>
> Either way, not a good spot to be in.
>
> Thoughts?
>
> --Udo
>

Re: [DISCUSS] Pulling the current proposed 1.10 release until we can agree on develop being stable

[Jacob Barrett <jb...@pivotal.io>]

While I share your concern that too many things are getting cherry-picked into the release, I disagree that recutting the branch is a good solution. Recutting the branch effectively cherry-picks everything on the develop branch. This means we start at ground zero evaluating this release. 

Each code change brings risk. We minimize that risk by reducing the change on the release branch. The alternative is to “lock” the develop branch, but this slows productivity significantly. 

I do think we could do a better job vetting what is and isn’t critical to the release though. I would like to see a harder line on new bugs vs. old bugs. Fixes to bugs introduced in this release are a clear candidate for including in the release. Bugs that have existed for several releases should probably get bumped with consideration given for likely or present impact to a user and complexity of the issue and fix.

-Jake


> On Aug 26, 2019, at 4:06 PM, Udo Kohlmeyer <ud...@apache.com> wrote:
> 
> Hi there Apache Geode devs,
> 
> It has been some weeks since the proposed 1.10 release was cut. We've gone through a few cycles where we keep on submitting "please include ticket GEODE-XXX" because it is critical and will break the system. WHICH in reality tells me that current develop is broken and unstable.
> 
> I'm going to suggest that we abandon the current 1.10 release branch. I cannot shake the feeling that our continuous cherry picking into a branch will result in either the branch becoming unmaintainable, given we have only select fixes in the branch OR we end up with a branch that is more stable than our current development branch, which would result in us having to rebase the develop branch onto the 1.10 branch.
> 
> I propose that once we see the pipeline is clean and green for a solid we then again attempt to cut 1.10 branch.
> 
> We CANNOT continue adding to a branch in order to stabilize it.. It just means the branch we cut from is unstable. If we cannot cut a branch from develop without having to have weeks of stabilization cycles, then our main branch is broken...
> 
> Either way, not a good spot to be in.
> 
> Thoughts?
> 
> --Udo
>