Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

Hi Dev@ List—

TL;DR:
I’d love it if we could modify the C* release cycle to include an additional “experimental” release branch that straddles the current major releases that includes somewhat “untested” or “risky” commits that normally would only go into the next major release. Releases based from this branch wouldn’t contain any features that require breaking changes or are considered highly “untested” or “risky” but would include the many other commits that today are considered too unsafe to put into the previous stable branch. This will allow us to run code closer to the current stable release branch when we are unable to move fully to the new major release branch. Also, during the release cycle of the next major release branch the project can get feedback from a subset of the total changes that will ultimately make it into that final new major release. Also — i’m aware that any additional branches/releases will add additional work for any developer that works on C*. It would be great if we could strike a balance that hopefully doesn’t add significant additional merging/rebasing/work for the team...

The Longer Story:
Last week I had a conversation with a few people regarding a proposed change to the current C* release schedule. 

Other than an attempt to make Jonathan and Sylvian’s lives more difficult, it would be ideal if we could better sync our internal release schedule with more recent Cassandra releases. The current cycle has resulted in currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential for another out-of-band 1.2/previous stable release build. We would love to always run the current “stable” release in production but generally/historically it takes time and a few minor releases to the current “major” branch stable to get to a state where we can accept for use in production. Additionally, as major releases are currently used to make “breaking” changes that require a more involved and risky upgrade process, it’s a much bigger deal to deploy a new major into production than a release without breaking changes. (upgrade-sstables for example is required when upgrading to a new major release branch. this unavoidable step adds lots of temporary load to the cluster and means deploying/upgrading to major releases tends to be a bit more risky than between minor releases and a more involved/long running process). This means even though there are months worth of stable hard work/awesome improvements in the current “stable” major release branch (today this is 2.0), we end up with an unavoidable and undesired lag in getting more recent C* changes pushed into production. This means we are unable to provide feedback on newer changes sooner to the community, stuck and unable to get even a subset of the awesome changes as we can’t yet take ALL the changes from the new major release branch, and finally if we find an issue in production or want to work on new functionality it would be ideal if we can write it against a release that is closer to the next major release while also providing us a reasonable way to get the feature deployed internally on a branch we are running.

Currently, the project generally tends to include all risky/breaking/more “feature” oriented tickets only into the next major release + trunk. However, there is a subset of these changes that are “somewhat” more risky changes but pose little/less/no risk the commit with introduce a regression outside of the scope of the patch/component. Additionally, any changes that  depend on other higher risk/breaking commits/changes wouldn’t be candidates for this proposed release branch. In a perfect world we would love to target a new “interim” or “experimental” train of releases which is loosely the most stable current release train but also includes a subset of changes from the next major train. (While we were discussing we thought about possible parallels to the concept of a LTS (Long Term Support) release cycle and what some people have dubbed the “tick-tock” release cycle.) This might look something like 1.2 branch + all moderately-to-“less”-risky/non-breaking commits which currently would only end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad candidates for this build would be for changes to components such as gossip, streaming, or any patch that changes the storage format etc). This would enable the project to provide builds for more active/risk-adverse users looking for a reasonable way to get more features and changes into production than with today’s release cycle. Additionally, this would hopefully facilitate/increase quicker feedback to the project on a subset of the new major release branch and any bugs found could be reported against an actual reproducible release instead of some custom build with a given number of patches from Jira or git SHAs applied/backported.

As it will always take both time and n releases to reach a stable minor release for a new major train; users could deploy this new release to get a subset of new features and changes with higher risk than would otherwise go into a minor release of the previous stable release train. If internally we wanted to add a new feature we could target this release while testing internally, and hopefully given the smaller delta between this “interim/experimental” to make it easier to re-base patches into the next major release train. This would help us avoid what today has unfortunately become a unavoidable large lag in getting new C* builds into production as while we attempt to sync our internal releases with a internally/or community QA’ed/accepted build/release of the current “stable” build/branch (currently this is 2.0).

To accomplish this, the commit workflow would unfortunately need change where an additional process is added to determine “eligibility” or “appropriateness” of a given commit to additionally also be committed to the “experimental” build branch (maybe it’s as simple as leaving it up to the reviewer + author to determine the risk factor and difficulty in merging the change back into the “experimental” build?). If it is agreed the commit/change/patch is a good candidate for the “experimental” branch, in addition to committing the patch to the current major release branch, the commit would also be merged into the new “experimental” release. If commits make it into the “experimental” branch frequently, I would expect/hope merging patches into the “experimental” build would be relatively easy as the “experimental” branch should also have most of the changes from the major release branch sans those considered highly risky or breaking. Additionally, if internally we want to work on a new feature and test internally before submitting a patch, we could target our code against the “experimental” branch, allowing us to test our changes in production without forking C* internally, writing our code against more recent “modern” changes, and then hopefully getting that work back to the community.


Hope this was clear enough and accurately summarizes the conversation a few of us had! Looking forward to everyone’s feedback and comments.

best,
kjellman

Re: Proposed changes to C* Release Schedule

[Jake Luciani <ja...@gmail.com>]

Hi Michael,

I didn't get to hear the in person conversation so taking a step back.
The proposal seems to be in response to a common problem.  i.e.  I'm on C*
version X and I need feature Y which is only available on version Z. Is
this correct?

The options have been: a) upgrade to version Z or b) fork C* and backport.
Coming my my previous job where I ran a prod C* cluster I felt this and I
expect many others do too.  We did have to fork and backport patches we
needed and it was hard.

This is specific to features and not bugs, since bugs are fixed in all
versions affected.

-Jake






On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
mkjellman@internalcircle.com> wrote:

> Hi Dev@ List—
>
> TL;DR:
> I’d love it if we could modify the C* release cycle to include an
> additional “experimental” release branch that straddles the current major
> releases that includes somewhat “untested” or “risky” commits that normally
> would only go into the next major release. Releases based from this branch
> wouldn’t contain any features that require breaking changes or are
> considered highly “untested” or “risky” but would include the many other
> commits that today are considered too unsafe to put into the previous
> stable branch. This will allow us to run code closer to the current stable
> release branch when we are unable to move fully to the new major release
> branch. Also, during the release cycle of the next major release branch the
> project can get feedback from a subset of the total changes that will
> ultimately make it into that final new major release. Also — i’m aware that
> any additional branches/releases will add additional work for any developer
> that works on C*. It would be great if we could strike a balance that
> hopefully doesn’t add significant additional merging/rebasing/work for the
> team...
>
> The Longer Story:
> Last week I had a conversation with a few people regarding a proposed
> change to the current C* release schedule.
>
> Other than an attempt to make Jonathan and Sylvian’s lives more difficult,
> it would be ideal if we could better sync our internal release schedule
> with more recent Cassandra releases. The current cycle has resulted in
> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable
> releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential for
> another out-of-band 1.2/previous stable release build. We would love to
> always run the current “stable” release in production but
> generally/historically it takes time and a few minor releases to the
> current “major” branch stable to get to a state where we can accept for use
> in production. Additionally, as major releases are currently used to make
> “breaking” changes that require a more involved and risky upgrade process,
> it’s a much bigger deal to deploy a new major into production than a
> release without breaking changes. (upgrade-sstables for example is required
> when upgrading to a new major release branch. this unavoidable step adds
> lots of temporary load to the cluster and means deploying/upgrading to
> major releases tends to be a bit more risky than between minor releases and
> a more involved/long running process). This means even though there are
> months worth of stable hard work/awesome improvements in the current
> “stable” major release branch (today this is 2.0), we end up with an
> unavoidable and undesired lag in getting more recent C* changes pushed into
> production. This means we are unable to provide feedback on newer changes
> sooner to the community, stuck and unable to get even a subset of the
> awesome changes as we can’t yet take ALL the changes from the new major
> release branch, and finally if we find an issue in production or want to
> work on new functionality it would be ideal if we can write it against a
> release that is closer to the next major release while also providing us a
> reasonable way to get the feature deployed internally on a branch we are
> running.
>
> Currently, the project generally tends to include all risky/breaking/more
> “feature” oriented tickets only into the next major release + trunk.
> However, there is a subset of these changes that are “somewhat” more risky
> changes but pose little/less/no risk the commit with introduce a regression
> outside of the scope of the patch/component. Additionally, any changes that
>  depend on other higher risk/breaking commits/changes wouldn’t be
> candidates for this proposed release branch. In a perfect world we would
> love to target a new “interim” or “experimental” train of releases which is
> loosely the most stable current release train but also includes a subset of
> changes from the next major train. (While we were discussing we thought
> about possible parallels to the concept of a LTS (Long Term Support)
> release cycle and what some people have dubbed the “tick-tock” release
> cycle.) This might look something like 1.2 branch + all
> moderately-to-“less”-risky/non-breaking commits which currently would only
> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad
> candidates for this build would be for changes to components such as
> gossip, streaming, or any patch that changes the storage format etc). This
> would enable the project to provide builds for more active/risk-adverse
> users looking for a reasonable way to get more features and changes into
> production than with today’s release cycle. Additionally, this would
> hopefully facilitate/increase quicker feedback to the project on a subset
> of the new major release branch and any bugs found could be reported
> against an actual reproducible release instead of some custom build with a
> given number of patches from Jira or git SHAs applied/backported.
>
> As it will always take both time and n releases to reach a stable minor
> release for a new major train; users could deploy this new release to get a
> subset of new features and changes with higher risk than would otherwise go
> into a minor release of the previous stable release train. If internally we
> wanted to add a new feature we could target this release while testing
> internally, and hopefully given the smaller delta between this
> “interim/experimental” to make it easier to re-base patches into the next
> major release train. This would help us avoid what today has unfortunately
> become a unavoidable large lag in getting new C* builds into production as
> while we attempt to sync our internal releases with a internally/or
> community QA’ed/accepted build/release of the current “stable” build/branch
> (currently this is 2.0).
>
> To accomplish this, the commit workflow would unfortunately need change
> where an additional process is added to determine “eligibility” or
> “appropriateness” of a given commit to additionally also be committed to
> the “experimental” build branch (maybe it’s as simple as leaving it up to
> the reviewer + author to determine the risk factor and difficulty in
> merging the change back into the “experimental” build?). If it is agreed
> the commit/change/patch is a good candidate for the “experimental” branch,
> in addition to committing the patch to the current major release branch,
> the commit would also be merged into the new “experimental” release. If
> commits make it into the “experimental” branch frequently, I would
> expect/hope merging patches into the “experimental” build would be
> relatively easy as the “experimental” branch should also have most of the
> changes from the major release branch sans those considered highly risky or
> breaking. Additionally, if internally we want to work on a new feature and
> test internally before submitting a patch, we could target our code against
> the “experimental” branch, allowing us to test our changes in production
> without forking C* internally, writing our code against more recent
> “modern” changes, and then hopefully getting that work back to the
> community.
>
>
> Hope this was clear enough and accurately summarizes the conversation a
> few of us had! Looking forward to everyone’s feedback and comments.
>
> best,
> kjellman




-- 
http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Jacob Rhoden <ja...@me.com>]

Isn't this how it works now? Aka

2.0 is the "I'm risk averse" stable, and
2.1 is the "I'm living on the edge" stable 

______________________________
Sent from iPhone

> On 17 Jun 2014, at 5:16 pm, Michael Kjellman <mk...@internalcircle.com> wrote:
> 
> Hi Dev@ List—
> 
> TL;DR:
> I’d love it if we could modify the C* release cycle to include an additional “experimental” release branch that straddles the current major releases that includes somewhat “untested” or “risky” commits that normally would only go into the next major release. Releases based from this branch wouldn’t contain any features that require breaking changes or are considered highly “untested” or “risky” but would include the many other commits that today are considered too unsafe to put into the previous stable branch. This will allow us to run code closer to the current stable release branch when we are unable to move fully to the new major release branch. Also, during the release cycle of the next major release branch the project can get feedback from a subset of the total changes that will ultimately make it into that final new major release. Also — i’m aware that any additional branches/releases will add additional work for any developer that works on C*. It would be great if we could strike a balance that hopefully doesn’t add significant additional merging/rebasing/work for the team...
> 
> The Longer Story:
> Last week I had a conversation with a few people regarding a proposed change to the current C* release schedule. 
> 
> Other than an attempt to make Jonathan and Sylvian’s lives more difficult, it would be ideal if we could better sync our internal release schedule with more recent Cassandra releases. The current cycle has resulted in currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential for another out-of-band 1.2/previous stable release build. We would love to always run the current “stable” release in production but generally/historically it takes time and a few minor releases to the current “major” branch stable to get to a state where we can accept for use in production. Additionally, as major releases are currently used to make “breaking” changes that require a more involved and risky upgrade process, it’s a much bigger deal to deploy a new major into production than a release without breaking changes. (upgrade-sstables for example is required when upgrading to a new major release branch. this unavoidable step adds lots of temporary load to the cluster and means deploying/upgrading to major releases tends to be a bit more risky than between minor releases and a more involved/long running process). This means even though there are months worth of stable hard work/awesome improvements in the current “stable” major release branch (today this is 2.0), we end up with an unavoidable and undesired lag in getting more recent C* changes pushed into production. This means we are unable to provide feedback on newer changes sooner to the community, stuck and unable to get even a subset of the awesome changes as we can’t yet take ALL the changes from the new major release branch, and finally if we find an issue in production or want to work on new functionality it would be ideal if we can write it against a release that is closer to the next major release while also providing us a reasonable way to get the feature deployed internally on a branch we are running.
> 
> Currently, the project generally tends to include all risky/breaking/more “feature” oriented tickets only into the next major release + trunk. However, there is a subset of these changes that are “somewhat” more risky changes but pose little/less/no risk the commit with introduce a regression outside of the scope of the patch/component. Additionally, any changes that  depend on other higher risk/breaking commits/changes wouldn’t be candidates for this proposed release branch. In a perfect world we would love to target a new “interim” or “experimental” train of releases which is loosely the most stable current release train but also includes a subset of changes from the next major train. (While we were discussing we thought about possible parallels to the concept of a LTS (Long Term Support) release cycle and what some people have dubbed the “tick-tock” release cycle.) This might look something like 1.2 branch + all moderately-to-“less”-risky/non-breaking commits which currently would only end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad candidates for this build would be for changes to components such as gossip, streaming, or any patch that changes the storage format etc). This would enable the project to provide builds for more active/risk-adverse users looking for a reasonable way to get more features and changes into production than with today’s release cycle. Additionally, this would hopefully facilitate/increase quicker feedback to the project on a subset of the new major release branch and any bugs found could be reported against an actual reproducible release instead of some custom build with a given number of patches from Jira or git SHAs applied/backported.
> 
> As it will always take both time and n releases to reach a stable minor release for a new major train; users could deploy this new release to get a subset of new features and changes with higher risk than would otherwise go into a minor release of the previous stable release train. If internally we wanted to add a new feature we could target this release while testing internally, and hopefully given the smaller delta between this “interim/experimental” to make it easier to re-base patches into the next major release train. This would help us avoid what today has unfortunately become a unavoidable large lag in getting new C* builds into production as while we attempt to sync our internal releases with a internally/or community QA’ed/accepted build/release of the current “stable” build/branch (currently this is 2.0).
> 
> To accomplish this, the commit workflow would unfortunately need change where an additional process is added to determine “eligibility” or “appropriateness” of a given commit to additionally also be committed to the “experimental” build branch (maybe it’s as simple as leaving it up to the reviewer + author to determine the risk factor and difficulty in merging the change back into the “experimental” build?). If it is agreed the commit/change/patch is a good candidate for the “experimental” branch, in addition to committing the patch to the current major release branch, the commit would also be merged into the new “experimental” release. If commits make it into the “experimental” branch frequently, I would expect/hope merging patches into the “experimental” build would be relatively easy as the “experimental” branch should also have most of the changes from the major release branch sans those considered highly risky or breaking. Additionally, if internally we want to work on a new feature and test internally before submitting a patch, we could target our code against the “experimental” branch, allowing us to test our changes in production without forking C* internally, writing our code against more recent “modern” changes, and then hopefully getting that work back to the community.
> 
> 
> Hope this was clear enough and accurately summarizes the conversation a few of us had! Looking forward to everyone’s feedback and comments.
> 
> best,
> kjellman

Re: Proposed changes to C* Release Schedule

[Brandon Williams <dr...@gmail.com>]

If that's what we want, merging is going to be much more painful.
 Currently we merge:

1.2->2.0->2.1->3.0

If we add an experimental branch for each, we still have to merge the
stable branch into experiemental:

1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex

And then the experimentals into each other:

1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex

That's quite a lot of merging in the end.


On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com> wrote:

> I'm not sure many people have the problem you are describing.  This is more
> of a C* developer issue than a C* user issue.
>
>
> Is the below what you are describing we move to?:
>
> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
>
> Specific changes would be backported based on the "less riskyness" of the
> change which you are assuming will be constant across versions?
>
> -Jake
>
>
>
>
> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
> mkjellman@internalcircle.com> wrote:
>
> > It's a bit about features - but it's more an attempt to achieve the goals
> > of what might happen with a 4 week release cycle (but that itself -- in
> > practice didn't prove to be valid/reasonable).
> >
> > If something like an executor service for performance is changed (for
> > example) it is definitely a more risky change than what would currently
> go
> > into 1.2 -- but most likely we would want to get patches like that into a
> > usable build.
> >
> > So I guess: a) reduce code drift between branches we run in production b)
> > get newer "features" into production faster where breaking changes aren't
> > required for the scope of the patch.
> >
> > Additionally - it's also a question of what release we use when we
> > identify an issue we want to work on internally. If we are on 1.2 because
> > we can't yet take ALL of 2.0 - do we now need to target our work against
> > 1.2? I would rather write it against the months worth of changes that
> have
> > happened since.
> >
> > Finally, it's an attempt to make the internal forking not as common as it
> > might be today. As you said - this is somewhat of a common process.
> >
> > > On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
> > >
> > > Hi Michael,
> > >
> > > I didn't get to hear the in person conversation so taking a step back.
> > > The proposal seems to be in response to a common problem.  i.e.  I'm on
> > C*
> > > version X and I need feature Y which is only available on version Z. Is
> > > this correct?
> > >
> > > The options have been: a) upgrade to version Z or b) fork C* and
> > backport.
> > > Coming my my previous job where I ran a prod C* cluster I felt this
> and I
> > > expect many others do too.  We did have to fork and backport patches we
> > > needed and it was hard.
> > >
> > > This is specific to features and not bugs, since bugs are fixed in all
> > > versions affected.
> > >
> > > -Jake
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> > > mkjellman@internalcircle.com> wrote:
> > >
> > >> Hi Dev@ List—
> > >>
> > >> TL;DR:
> > >> I’d love it if we could modify the C* release cycle to include an
> > >> additional “experimental” release branch that straddles the current
> > major
> > >> releases that includes somewhat “untested” or “risky” commits that
> > normally
> > >> would only go into the next major release. Releases based from this
> > branch
> > >> wouldn’t contain any features that require breaking changes or are
> > >> considered highly “untested” or “risky” but would include the many
> other
> > >> commits that today are considered too unsafe to put into the previous
> > >> stable branch. This will allow us to run code closer to the current
> > stable
> > >> release branch when we are unable to move fully to the new major
> release
> > >> branch. Also, during the release cycle of the next major release
> branch
> > the
> > >> project can get feedback from a subset of the total changes that will
> > >> ultimately make it into that final new major release. Also — i’m aware
> > that
> > >> any additional branches/releases will add additional work for any
> > developer
> > >> that works on C*. It would be great if we could strike a balance that
> > >> hopefully doesn’t add significant additional merging/rebasing/work for
> > the
> > >> team...
> > >>
> > >> The Longer Story:
> > >> Last week I had a conversation with a few people regarding a proposed
> > >> change to the current C* release schedule.
> > >>
> > >> Other than an attempt to make Jonathan and Sylvian’s lives more
> > difficult,
> > >> it would be ideal if we could better sync our internal release
> schedule
> > >> with more recent Cassandra releases. The current cycle has resulted in
> > >> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
> stable
> > >> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
> potential
> > for
> > >> another out-of-band 1.2/previous stable release build. We would love
> to
> > >> always run the current “stable” release in production but
> > >> generally/historically it takes time and a few minor releases to the
> > >> current “major” branch stable to get to a state where we can accept
> for
> > use
> > >> in production. Additionally, as major releases are currently used to
> > make
> > >> “breaking” changes that require a more involved and risky upgrade
> > process,
> > >> it’s a much bigger deal to deploy a new major into production than a
> > >> release without breaking changes. (upgrade-sstables for example is
> > required
> > >> when upgrading to a new major release branch. this unavoidable step
> adds
> > >> lots of temporary load to the cluster and means deploying/upgrading to
> > >> major releases tends to be a bit more risky than between minor
> releases
> > and
> > >> a more involved/long running process). This means even though there
> are
> > >> months worth of stable hard work/awesome improvements in the current
> > >> “stable” major release branch (today this is 2.0), we end up with an
> > >> unavoidable and undesired lag in getting more recent C* changes pushed
> > into
> > >> production. This means we are unable to provide feedback on newer
> > changes
> > >> sooner to the community, stuck and unable to get even a subset of the
> > >> awesome changes as we can’t yet take ALL the changes from the new
> major
> > >> release branch, and finally if we find an issue in production or want
> to
> > >> work on new functionality it would be ideal if we can write it
> against a
> > >> release that is closer to the next major release while also providing
> > us a
> > >> reasonable way to get the feature deployed internally on a branch we
> are
> > >> running.
> > >>
> > >> Currently, the project generally tends to include all
> > risky/breaking/more
> > >> “feature” oriented tickets only into the next major release + trunk.
> > >> However, there is a subset of these changes that are “somewhat” more
> > risky
> > >> changes but pose little/less/no risk the commit with introduce a
> > regression
> > >> outside of the scope of the patch/component. Additionally, any changes
> > that
> > >> depend on other higher risk/breaking commits/changes wouldn’t be
> > >> candidates for this proposed release branch. In a perfect world we
> would
> > >> love to target a new “interim” or “experimental” train of releases
> > which is
> > >> loosely the most stable current release train but also includes a
> > subset of
> > >> changes from the next major train. (While we were discussing we
> thought
> > >> about possible parallels to the concept of a LTS (Long Term Support)
> > >> release cycle and what some people have dubbed the “tick-tock” release
> > >> cycle.) This might look something like 1.2 branch + all
> > >> moderately-to-“less”-risky/non-breaking commits which currently would
> > only
> > >> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately
> bad
> > >> candidates for this build would be for changes to components such as
> > >> gossip, streaming, or any patch that changes the storage format etc).
> > This
> > >> would enable the project to provide builds for more
> active/risk-adverse
> > >> users looking for a reasonable way to get more features and changes
> into
> > >> production than with today’s release cycle. Additionally, this would
> > >> hopefully facilitate/increase quicker feedback to the project on a
> > subset
> > >> of the new major release branch and any bugs found could be reported
> > >> against an actual reproducible release instead of some custom build
> > with a
> > >> given number of patches from Jira or git SHAs applied/backported.
> > >>
> > >> As it will always take both time and n releases to reach a stable
> minor
> > >> release for a new major train; users could deploy this new release to
> > get a
> > >> subset of new features and changes with higher risk than would
> > otherwise go
> > >> into a minor release of the previous stable release train. If
> > internally we
> > >> wanted to add a new feature we could target this release while testing
> > >> internally, and hopefully given the smaller delta between this
> > >> “interim/experimental” to make it easier to re-base patches into the
> > next
> > >> major release train. This would help us avoid what today has
> > unfortunately
> > >> become a unavoidable large lag in getting new C* builds into
> production
> > as
> > >> while we attempt to sync our internal releases with a internally/or
> > >> community QA’ed/accepted build/release of the current “stable”
> > build/branch
> > >> (currently this is 2.0).
> > >>
> > >> To accomplish this, the commit workflow would unfortunately need
> change
> > >> where an additional process is added to determine “eligibility” or
> > >> “appropriateness” of a given commit to additionally also be committed
> to
> > >> the “experimental” build branch (maybe it’s as simple as leaving it up
> > to
> > >> the reviewer + author to determine the risk factor and difficulty in
> > >> merging the change back into the “experimental” build?). If it is
> agreed
> > >> the commit/change/patch is a good candidate for the “experimental”
> > branch,
> > >> in addition to committing the patch to the current major release
> branch,
> > >> the commit would also be merged into the new “experimental” release.
> If
> > >> commits make it into the “experimental” branch frequently, I would
> > >> expect/hope merging patches into the “experimental” build would be
> > >> relatively easy as the “experimental” branch should also have most of
> > the
> > >> changes from the major release branch sans those considered highly
> > risky or
> > >> breaking. Additionally, if internally we want to work on a new feature
> > and
> > >> test internally before submitting a patch, we could target our code
> > against
> > >> the “experimental” branch, allowing us to test our changes in
> production
> > >> without forking C* internally, writing our code against more recent
> > >> “modern” changes, and then hopefully getting that work back to the
> > >> community.
> > >>
> > >>
> > >> Hope this was clear enough and accurately summarizes the conversation
> a
> > >> few of us had! Looking forward to everyone’s feedback and comments.
> > >>
> > >> best,
> > >> kjellman
> > >
> > >
> > >
> > >
> > > --
> > > http://twitter.com/tjake
> >
>
>
>
> --
> http://twitter.com/tjake
>

Re: Proposed changes to C* Release Schedule

[Jason Brown <ja...@gmail.com>]

FWIW, here's the ubuntu wiki on their release model:
https://wiki.ubuntu.com/TimeBasedReleases

Short story: they have a feature set for each release, and it must be
tested thoroughly before going out the door. Granted, they are shipping an
entire OS with 1000s of components that all must work well *together*, but
there may be something we can learn from their process - I'm not
necessarily recommending we adopt it, but it seems to work for Ubuntu.

-Jason



On Tue, Jun 17, 2014 at 10:27 AM, Jake Luciani <ja...@gmail.com> wrote:

> So there are two issues this proposal is trying to address:
>
> 1. Shrink the release cycle.
>
> 2. Backport things to stable releases.
>
> We should discuss these separately since together it's hard to discuss.
> 1. is less controversial I would think :)
>
>
>
>
>
>
> On Tue, Jun 17, 2014 at 1:16 PM, Michael Kjellman <
> mkjellman@internalcircle.com> wrote:
>
> > Totally agree — "Also — i’m aware that any additional branches/releases
> > will add additional work for any developer that works on C*. It would be
> > great if we could strike a balance that hopefully doesn’t add significant
> > additional merging/rebasing/work for the team…”
> >
> > That being said I don’t think i’m alone by identifying the problem. The
> > proposed solution was what we came up with in the hour or so we discussed
> > this in person. How else can you shrink the release schedule without
> > creating another branch? Also — the idea is to only have this branch
> > “active” during the overlap when major release branches need to
> stabilize.
> >
> > > On Jun 17, 2014, at 10:03 AM, Brandon Williams <dr...@gmail.com>
> wrote:
> > >
> > > If that's what we want, merging is going to be much more painful.
> > > Currently we merge:
> > >
> > > 1.2->2.0->2.1->3.0
> > >
> > > If we add an experimental branch for each, we still have to merge the
> > > stable branch into experiemental:
> > >
> > > 1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex
> > >
> > > And then the experimentals into each other:
> > >
> > > 1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex
> > >
> > > That's quite a lot of merging in the end.
> > >
> > >
> > > On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com>
> wrote:
> > >
> > >> I'm not sure many people have the problem you are describing.  This is
> > more
> > >> of a C* developer issue than a C* user issue.
> > >>
> > >>
> > >> Is the below what you are describing we move to?:
> > >>
> > >> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
> > >> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
> > >>
> > >> Specific changes would be backported based on the "less riskyness" of
> > the
> > >> change which you are assuming will be constant across versions?
> > >>
> > >> -Jake
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
> > >> mkjellman@internalcircle.com> wrote:
> > >>
> > >>> It's a bit about features - but it's more an attempt to achieve the
> > goals
> > >>> of what might happen with a 4 week release cycle (but that itself --
> in
> > >>> practice didn't prove to be valid/reasonable).
> > >>>
> > >>> If something like an executor service for performance is changed (for
> > >>> example) it is definitely a more risky change than what would
> currently
> > >> go
> > >>> into 1.2 -- but most likely we would want to get patches like that
> > into a
> > >>> usable build.
> > >>>
> > >>> So I guess: a) reduce code drift between branches we run in
> production
> > b)
> > >>> get newer "features" into production faster where breaking changes
> > aren't
> > >>> required for the scope of the patch.
> > >>>
> > >>> Additionally - it's also a question of what release we use when we
> > >>> identify an issue we want to work on internally. If we are on 1.2
> > because
> > >>> we can't yet take ALL of 2.0 - do we now need to target our work
> > against
> > >>> 1.2? I would rather write it against the months worth of changes that
> > >> have
> > >>> happened since.
> > >>>
> > >>> Finally, it's an attempt to make the internal forking not as common
> as
> > it
> > >>> might be today. As you said - this is somewhat of a common process.
> > >>>
> > >>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com>
> wrote
> > >>>>
> > >>>> Hi Michael,
> > >>>>
> > >>>> I didn't get to hear the in person conversation so taking a step
> back.
> > >>>> The proposal seems to be in response to a common problem.  i.e.  I'm
> > on
> > >>> C*
> > >>>> version X and I need feature Y which is only available on version Z.
> > Is
> > >>>> this correct?
> > >>>>
> > >>>> The options have been: a) upgrade to version Z or b) fork C* and
> > >>> backport.
> > >>>> Coming my my previous job where I ran a prod C* cluster I felt this
> > >> and I
> > >>>> expect many others do too.  We did have to fork and backport patches
> > we
> > >>>> needed and it was hard.
> > >>>>
> > >>>> This is specific to features and not bugs, since bugs are fixed in
> all
> > >>>> versions affected.
> > >>>>
> > >>>> -Jake
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> > >>>> mkjellman@internalcircle.com> wrote:
> > >>>>
> > >>>>> Hi Dev@ List—
> > >>>>>
> > >>>>> TL;DR:
> > >>>>> I’d love it if we could modify the C* release cycle to include an
> > >>>>> additional “experimental” release branch that straddles the current
> > >>> major
> > >>>>> releases that includes somewhat “untested” or “risky” commits that
> > >>> normally
> > >>>>> would only go into the next major release. Releases based from this
> > >>> branch
> > >>>>> wouldn’t contain any features that require breaking changes or are
> > >>>>> considered highly “untested” or “risky” but would include the many
> > >> other
> > >>>>> commits that today are considered too unsafe to put into the
> previous
> > >>>>> stable branch. This will allow us to run code closer to the current
> > >>> stable
> > >>>>> release branch when we are unable to move fully to the new major
> > >> release
> > >>>>> branch. Also, during the release cycle of the next major release
> > >> branch
> > >>> the
> > >>>>> project can get feedback from a subset of the total changes that
> will
> > >>>>> ultimately make it into that final new major release. Also — i’m
> > aware
> > >>> that
> > >>>>> any additional branches/releases will add additional work for any
> > >>> developer
> > >>>>> that works on C*. It would be great if we could strike a balance
> that
> > >>>>> hopefully doesn’t add significant additional merging/rebasing/work
> > for
> > >>> the
> > >>>>> team...
> > >>>>>
> > >>>>> The Longer Story:
> > >>>>> Last week I had a conversation with a few people regarding a
> proposed
> > >>>>> change to the current C* release schedule.
> > >>>>>
> > >>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
> > >>> difficult,
> > >>>>> it would be ideal if we could better sync our internal release
> > >> schedule
> > >>>>> with more recent Cassandra releases. The current cycle has resulted
> > in
> > >>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
> > >> stable
> > >>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
> > >> potential
> > >>> for
> > >>>>> another out-of-band 1.2/previous stable release build. We would
> love
> > >> to
> > >>>>> always run the current “stable” release in production but
> > >>>>> generally/historically it takes time and a few minor releases to
> the
> > >>>>> current “major” branch stable to get to a state where we can accept
> > >> for
> > >>> use
> > >>>>> in production. Additionally, as major releases are currently used
> to
> > >>> make
> > >>>>> “breaking” changes that require a more involved and risky upgrade
> > >>> process,
> > >>>>> it’s a much bigger deal to deploy a new major into production than
> a
> > >>>>> release without breaking changes. (upgrade-sstables for example is
> > >>> required
> > >>>>> when upgrading to a new major release branch. this unavoidable step
> > >> adds
> > >>>>> lots of temporary load to the cluster and means deploying/upgrading
> > to
> > >>>>> major releases tends to be a bit more risky than between minor
> > >> releases
> > >>> and
> > >>>>> a more involved/long running process). This means even though there
> > >> are
> > >>>>> months worth of stable hard work/awesome improvements in the
> current
> > >>>>> “stable” major release branch (today this is 2.0), we end up with
> an
> > >>>>> unavoidable and undesired lag in getting more recent C* changes
> > pushed
> > >>> into
> > >>>>> production. This means we are unable to provide feedback on newer
> > >>> changes
> > >>>>> sooner to the community, stuck and unable to get even a subset of
> the
> > >>>>> awesome changes as we can’t yet take ALL the changes from the new
> > >> major
> > >>>>> release branch, and finally if we find an issue in production or
> want
> > >> to
> > >>>>> work on new functionality it would be ideal if we can write it
> > >> against a
> > >>>>> release that is closer to the next major release while also
> providing
> > >>> us a
> > >>>>> reasonable way to get the feature deployed internally on a branch
> we
> > >> are
> > >>>>> running.
> > >>>>>
> > >>>>> Currently, the project generally tends to include all
> > >>> risky/breaking/more
> > >>>>> “feature” oriented tickets only into the next major release +
> trunk.
> > >>>>> However, there is a subset of these changes that are “somewhat”
> more
> > >>> risky
> > >>>>> changes but pose little/less/no risk the commit with introduce a
> > >>> regression
> > >>>>> outside of the scope of the patch/component. Additionally, any
> > changes
> > >>> that
> > >>>>> depend on other higher risk/breaking commits/changes wouldn’t be
> > >>>>> candidates for this proposed release branch. In a perfect world we
> > >> would
> > >>>>> love to target a new “interim” or “experimental” train of releases
> > >>> which is
> > >>>>> loosely the most stable current release train but also includes a
> > >>> subset of
> > >>>>> changes from the next major train. (While we were discussing we
> > >> thought
> > >>>>> about possible parallels to the concept of a LTS (Long Term
> Support)
> > >>>>> release cycle and what some people have dubbed the “tick-tock”
> > release
> > >>>>> cycle.) This might look something like 1.2 branch + all
> > >>>>> moderately-to-“less”-risky/non-breaking commits which currently
> would
> > >>> only
> > >>>>> end up in a 2.0 or 2.1 release. (Off the top of my head,
> immediately
> > >> bad
> > >>>>> candidates for this build would be for changes to components such
> as
> > >>>>> gossip, streaming, or any patch that changes the storage format
> etc).
> > >>> This
> > >>>>> would enable the project to provide builds for more
> > >> active/risk-adverse
> > >>>>> users looking for a reasonable way to get more features and changes
> > >> into
> > >>>>> production than with today’s release cycle. Additionally, this
> would
> > >>>>> hopefully facilitate/increase quicker feedback to the project on a
> > >>> subset
> > >>>>> of the new major release branch and any bugs found could be
> reported
> > >>>>> against an actual reproducible release instead of some custom build
> > >>> with a
> > >>>>> given number of patches from Jira or git SHAs applied/backported.
> > >>>>>
> > >>>>> As it will always take both time and n releases to reach a stable
> > >> minor
> > >>>>> release for a new major train; users could deploy this new release
> to
> > >>> get a
> > >>>>> subset of new features and changes with higher risk than would
> > >>> otherwise go
> > >>>>> into a minor release of the previous stable release train. If
> > >>> internally we
> > >>>>> wanted to add a new feature we could target this release while
> > testing
> > >>>>> internally, and hopefully given the smaller delta between this
> > >>>>> “interim/experimental” to make it easier to re-base patches into
> the
> > >>> next
> > >>>>> major release train. This would help us avoid what today has
> > >>> unfortunately
> > >>>>> become a unavoidable large lag in getting new C* builds into
> > >> production
> > >>> as
> > >>>>> while we attempt to sync our internal releases with a internally/or
> > >>>>> community QA’ed/accepted build/release of the current “stable”
> > >>> build/branch
> > >>>>> (currently this is 2.0).
> > >>>>>
> > >>>>> To accomplish this, the commit workflow would unfortunately need
> > >> change
> > >>>>> where an additional process is added to determine “eligibility” or
> > >>>>> “appropriateness” of a given commit to additionally also be
> committed
> > >> to
> > >>>>> the “experimental” build branch (maybe it’s as simple as leaving it
> > up
> > >>> to
> > >>>>> the reviewer + author to determine the risk factor and difficulty
> in
> > >>>>> merging the change back into the “experimental” build?). If it is
> > >> agreed
> > >>>>> the commit/change/patch is a good candidate for the “experimental”
> > >>> branch,
> > >>>>> in addition to committing the patch to the current major release
> > >> branch,
> > >>>>> the commit would also be merged into the new “experimental”
> release.
> > >> If
> > >>>>> commits make it into the “experimental” branch frequently, I would
> > >>>>> expect/hope merging patches into the “experimental” build would be
> > >>>>> relatively easy as the “experimental” branch should also have most
> of
> > >>> the
> > >>>>> changes from the major release branch sans those considered highly
> > >>> risky or
> > >>>>> breaking. Additionally, if internally we want to work on a new
> > feature
> > >>> and
> > >>>>> test internally before submitting a patch, we could target our code
> > >>> against
> > >>>>> the “experimental” branch, allowing us to test our changes in
> > >> production
> > >>>>> without forking C* internally, writing our code against more recent
> > >>>>> “modern” changes, and then hopefully getting that work back to the
> > >>>>> community.
> > >>>>>
> > >>>>>
> > >>>>> Hope this was clear enough and accurately summarizes the
> conversation
> > >> a
> > >>>>> few of us had! Looking forward to everyone’s feedback and comments.
> > >>>>>
> > >>>>> best,
> > >>>>> kjellman
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> http://twitter.com/tjake
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> http://twitter.com/tjake
> > >>
> >
> >
>
>
> --
> http://twitter.com/tjake
>

Re: Proposed changes to C* Release Schedule

[Jake Luciani <ja...@gmail.com>]

So there are two issues this proposal is trying to address:

1. Shrink the release cycle.

2. Backport things to stable releases.

We should discuss these separately since together it's hard to discuss.
1. is less controversial I would think :)






On Tue, Jun 17, 2014 at 1:16 PM, Michael Kjellman <
mkjellman@internalcircle.com> wrote:

> Totally agree — "Also — i’m aware that any additional branches/releases
> will add additional work for any developer that works on C*. It would be
> great if we could strike a balance that hopefully doesn’t add significant
> additional merging/rebasing/work for the team…”
>
> That being said I don’t think i’m alone by identifying the problem. The
> proposed solution was what we came up with in the hour or so we discussed
> this in person. How else can you shrink the release schedule without
> creating another branch? Also — the idea is to only have this branch
> “active” during the overlap when major release branches need to stabilize.
>
> > On Jun 17, 2014, at 10:03 AM, Brandon Williams <dr...@gmail.com> wrote:
> >
> > If that's what we want, merging is going to be much more painful.
> > Currently we merge:
> >
> > 1.2->2.0->2.1->3.0
> >
> > If we add an experimental branch for each, we still have to merge the
> > stable branch into experiemental:
> >
> > 1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex
> >
> > And then the experimentals into each other:
> >
> > 1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex
> >
> > That's quite a lot of merging in the end.
> >
> >
> > On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com> wrote:
> >
> >> I'm not sure many people have the problem you are describing.  This is
> more
> >> of a C* developer issue than a C* user issue.
> >>
> >>
> >> Is the below what you are describing we move to?:
> >>
> >> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
> >> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
> >>
> >> Specific changes would be backported based on the "less riskyness" of
> the
> >> change which you are assuming will be constant across versions?
> >>
> >> -Jake
> >>
> >>
> >>
> >>
> >> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
> >> mkjellman@internalcircle.com> wrote:
> >>
> >>> It's a bit about features - but it's more an attempt to achieve the
> goals
> >>> of what might happen with a 4 week release cycle (but that itself -- in
> >>> practice didn't prove to be valid/reasonable).
> >>>
> >>> If something like an executor service for performance is changed (for
> >>> example) it is definitely a more risky change than what would currently
> >> go
> >>> into 1.2 -- but most likely we would want to get patches like that
> into a
> >>> usable build.
> >>>
> >>> So I guess: a) reduce code drift between branches we run in production
> b)
> >>> get newer "features" into production faster where breaking changes
> aren't
> >>> required for the scope of the patch.
> >>>
> >>> Additionally - it's also a question of what release we use when we
> >>> identify an issue we want to work on internally. If we are on 1.2
> because
> >>> we can't yet take ALL of 2.0 - do we now need to target our work
> against
> >>> 1.2? I would rather write it against the months worth of changes that
> >> have
> >>> happened since.
> >>>
> >>> Finally, it's an attempt to make the internal forking not as common as
> it
> >>> might be today. As you said - this is somewhat of a common process.
> >>>
> >>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
> >>>>
> >>>> Hi Michael,
> >>>>
> >>>> I didn't get to hear the in person conversation so taking a step back.
> >>>> The proposal seems to be in response to a common problem.  i.e.  I'm
> on
> >>> C*
> >>>> version X and I need feature Y which is only available on version Z.
> Is
> >>>> this correct?
> >>>>
> >>>> The options have been: a) upgrade to version Z or b) fork C* and
> >>> backport.
> >>>> Coming my my previous job where I ran a prod C* cluster I felt this
> >> and I
> >>>> expect many others do too.  We did have to fork and backport patches
> we
> >>>> needed and it was hard.
> >>>>
> >>>> This is specific to features and not bugs, since bugs are fixed in all
> >>>> versions affected.
> >>>>
> >>>> -Jake
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> >>>> mkjellman@internalcircle.com> wrote:
> >>>>
> >>>>> Hi Dev@ List—
> >>>>>
> >>>>> TL;DR:
> >>>>> I’d love it if we could modify the C* release cycle to include an
> >>>>> additional “experimental” release branch that straddles the current
> >>> major
> >>>>> releases that includes somewhat “untested” or “risky” commits that
> >>> normally
> >>>>> would only go into the next major release. Releases based from this
> >>> branch
> >>>>> wouldn’t contain any features that require breaking changes or are
> >>>>> considered highly “untested” or “risky” but would include the many
> >> other
> >>>>> commits that today are considered too unsafe to put into the previous
> >>>>> stable branch. This will allow us to run code closer to the current
> >>> stable
> >>>>> release branch when we are unable to move fully to the new major
> >> release
> >>>>> branch. Also, during the release cycle of the next major release
> >> branch
> >>> the
> >>>>> project can get feedback from a subset of the total changes that will
> >>>>> ultimately make it into that final new major release. Also — i’m
> aware
> >>> that
> >>>>> any additional branches/releases will add additional work for any
> >>> developer
> >>>>> that works on C*. It would be great if we could strike a balance that
> >>>>> hopefully doesn’t add significant additional merging/rebasing/work
> for
> >>> the
> >>>>> team...
> >>>>>
> >>>>> The Longer Story:
> >>>>> Last week I had a conversation with a few people regarding a proposed
> >>>>> change to the current C* release schedule.
> >>>>>
> >>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
> >>> difficult,
> >>>>> it would be ideal if we could better sync our internal release
> >> schedule
> >>>>> with more recent Cassandra releases. The current cycle has resulted
> in
> >>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
> >> stable
> >>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
> >> potential
> >>> for
> >>>>> another out-of-band 1.2/previous stable release build. We would love
> >> to
> >>>>> always run the current “stable” release in production but
> >>>>> generally/historically it takes time and a few minor releases to the
> >>>>> current “major” branch stable to get to a state where we can accept
> >> for
> >>> use
> >>>>> in production. Additionally, as major releases are currently used to
> >>> make
> >>>>> “breaking” changes that require a more involved and risky upgrade
> >>> process,
> >>>>> it’s a much bigger deal to deploy a new major into production than a
> >>>>> release without breaking changes. (upgrade-sstables for example is
> >>> required
> >>>>> when upgrading to a new major release branch. this unavoidable step
> >> adds
> >>>>> lots of temporary load to the cluster and means deploying/upgrading
> to
> >>>>> major releases tends to be a bit more risky than between minor
> >> releases
> >>> and
> >>>>> a more involved/long running process). This means even though there
> >> are
> >>>>> months worth of stable hard work/awesome improvements in the current
> >>>>> “stable” major release branch (today this is 2.0), we end up with an
> >>>>> unavoidable and undesired lag in getting more recent C* changes
> pushed
> >>> into
> >>>>> production. This means we are unable to provide feedback on newer
> >>> changes
> >>>>> sooner to the community, stuck and unable to get even a subset of the
> >>>>> awesome changes as we can’t yet take ALL the changes from the new
> >> major
> >>>>> release branch, and finally if we find an issue in production or want
> >> to
> >>>>> work on new functionality it would be ideal if we can write it
> >> against a
> >>>>> release that is closer to the next major release while also providing
> >>> us a
> >>>>> reasonable way to get the feature deployed internally on a branch we
> >> are
> >>>>> running.
> >>>>>
> >>>>> Currently, the project generally tends to include all
> >>> risky/breaking/more
> >>>>> “feature” oriented tickets only into the next major release + trunk.
> >>>>> However, there is a subset of these changes that are “somewhat” more
> >>> risky
> >>>>> changes but pose little/less/no risk the commit with introduce a
> >>> regression
> >>>>> outside of the scope of the patch/component. Additionally, any
> changes
> >>> that
> >>>>> depend on other higher risk/breaking commits/changes wouldn’t be
> >>>>> candidates for this proposed release branch. In a perfect world we
> >> would
> >>>>> love to target a new “interim” or “experimental” train of releases
> >>> which is
> >>>>> loosely the most stable current release train but also includes a
> >>> subset of
> >>>>> changes from the next major train. (While we were discussing we
> >> thought
> >>>>> about possible parallels to the concept of a LTS (Long Term Support)
> >>>>> release cycle and what some people have dubbed the “tick-tock”
> release
> >>>>> cycle.) This might look something like 1.2 branch + all
> >>>>> moderately-to-“less”-risky/non-breaking commits which currently would
> >>> only
> >>>>> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately
> >> bad
> >>>>> candidates for this build would be for changes to components such as
> >>>>> gossip, streaming, or any patch that changes the storage format etc).
> >>> This
> >>>>> would enable the project to provide builds for more
> >> active/risk-adverse
> >>>>> users looking for a reasonable way to get more features and changes
> >> into
> >>>>> production than with today’s release cycle. Additionally, this would
> >>>>> hopefully facilitate/increase quicker feedback to the project on a
> >>> subset
> >>>>> of the new major release branch and any bugs found could be reported
> >>>>> against an actual reproducible release instead of some custom build
> >>> with a
> >>>>> given number of patches from Jira or git SHAs applied/backported.
> >>>>>
> >>>>> As it will always take both time and n releases to reach a stable
> >> minor
> >>>>> release for a new major train; users could deploy this new release to
> >>> get a
> >>>>> subset of new features and changes with higher risk than would
> >>> otherwise go
> >>>>> into a minor release of the previous stable release train. If
> >>> internally we
> >>>>> wanted to add a new feature we could target this release while
> testing
> >>>>> internally, and hopefully given the smaller delta between this
> >>>>> “interim/experimental” to make it easier to re-base patches into the
> >>> next
> >>>>> major release train. This would help us avoid what today has
> >>> unfortunately
> >>>>> become a unavoidable large lag in getting new C* builds into
> >> production
> >>> as
> >>>>> while we attempt to sync our internal releases with a internally/or
> >>>>> community QA’ed/accepted build/release of the current “stable”
> >>> build/branch
> >>>>> (currently this is 2.0).
> >>>>>
> >>>>> To accomplish this, the commit workflow would unfortunately need
> >> change
> >>>>> where an additional process is added to determine “eligibility” or
> >>>>> “appropriateness” of a given commit to additionally also be committed
> >> to
> >>>>> the “experimental” build branch (maybe it’s as simple as leaving it
> up
> >>> to
> >>>>> the reviewer + author to determine the risk factor and difficulty in
> >>>>> merging the change back into the “experimental” build?). If it is
> >> agreed
> >>>>> the commit/change/patch is a good candidate for the “experimental”
> >>> branch,
> >>>>> in addition to committing the patch to the current major release
> >> branch,
> >>>>> the commit would also be merged into the new “experimental” release.
> >> If
> >>>>> commits make it into the “experimental” branch frequently, I would
> >>>>> expect/hope merging patches into the “experimental” build would be
> >>>>> relatively easy as the “experimental” branch should also have most of
> >>> the
> >>>>> changes from the major release branch sans those considered highly
> >>> risky or
> >>>>> breaking. Additionally, if internally we want to work on a new
> feature
> >>> and
> >>>>> test internally before submitting a patch, we could target our code
> >>> against
> >>>>> the “experimental” branch, allowing us to test our changes in
> >> production
> >>>>> without forking C* internally, writing our code against more recent
> >>>>> “modern” changes, and then hopefully getting that work back to the
> >>>>> community.
> >>>>>
> >>>>>
> >>>>> Hope this was clear enough and accurately summarizes the conversation
> >> a
> >>>>> few of us had! Looking forward to everyone’s feedback and comments.
> >>>>>
> >>>>> best,
> >>>>> kjellman
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://twitter.com/tjake
> >>>
> >>
> >>
> >>
> >> --
> >> http://twitter.com/tjake
> >>
>
>


-- 
http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Sylvain Lebresne <sy...@datastax.com>]

On Tue, Jun 24, 2014 at 5:26 PM, Jonathan Ellis <jb...@gmail.com> wrote:

>
> What if we tried a quicker release cycle, BUT we would guarantee that
> you could do a rolling upgrade until we bump the supermajor version?
> So 2.0 could upgrade to 3.0 without having to go through 2.1.  (But to
> go to 3.1 or 4.0 you would have to go through 3.0.)
>

I was thinking of something along those lines so I'm in favor of giving
that a try.

More precisely, I was thinking we could lower the release cycle to 4 month
(less
feels hard to achieve) but make a "supermajor" only every 2 releases (or
less
often, though "guarantee that you could do a rolling upgrade" imply that we
rigorously
test that and I think aiming for 2 release in a row initially is a good
start).

It's worth acknowledging that this will probably involve a tad more merging
work
but it feels that increase might be reasonable.

--
Sylvain


> On Tue, Jun 24, 2014 at 8:27 AM, Chris Burroughs
> <ch...@gmail.com> wrote:
> > On 06/17/2014 01:16 PM, Michael Kjellman wrote:
> >>
> >> That being said I don’t think i’m alone by identifying the problem.
> >
> >
> > FWIW I'm not doing anything wildly unusual and I've been on a fork for as
> > long as I've been on 1.2 (and various times before).  Almost everyone
> being
> > on 1.2 with 3 other equally weighted active branches seems like an
> obvious
> > not-great situation for running cassandra or developing.
> >
> > I like the idea of shortening the release cycle and LTS style releases
> and
> > they feel like the most direct approach.  I'm a little wary of more
> branches
> > since that could backfire and make the problem worse.
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

Humm — sorry guys — I never got Chris or Jonathan’s responses for some reason.

That being said, sounds like a good compromise Sylvain. Fingers crossed this turns into a good experiment! Thanks

best,
kjellman


> On Jun 24, 2014, at 10:09 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> 
> On Tue, Jun 24, 2014 at 5:26 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> 
>> 
>> What if we tried a quicker release cycle, BUT we would guarantee that
>> you could do a rolling upgrade until we bump the supermajor version?
>> So 2.0 could upgrade to 3.0 without having to go through 2.1.  (But to
>> go to 3.1 or 4.0 you would have to go through 3.0.)
>> 
> 
> I was thinking of something along those lines so I'm in favor of giving
> that a try.
> 
> More precisely, I was thinking we could lower the release cycle to 4 month
> (less
> feels hard to achieve) but make a "supermajor" only every 2 releases (or
> less
> often, though "guarantee that you could do a rolling upgrade" imply that we
> rigorously
> test that and I think aiming for 2 release in a row initially is a good
> start).
> 
> It's worth acknowledging that this will probably involve a tad more merging
> work
> but it feels that increase might be reasonable.
> 
> --
> Sylvain
> 
> 
>> On Tue, Jun 24, 2014 at 8:27 AM, Chris Burroughs
>> <ch...@gmail.com> wrote:
>>> On 06/17/2014 01:16 PM, Michael Kjellman wrote:
>>>> 
>>>> That being said I don’t think i’m alone by identifying the problem.
>>> 
>>> 
>>> FWIW I'm not doing anything wildly unusual and I've been on a fork for as
>>> long as I've been on 1.2 (and various times before).  Almost everyone
>> being
>>> on 1.2 with 3 other equally weighted active branches seems like an
>> obvious
>>> not-great situation for running cassandra or developing.
>>> 
>>> I like the idea of shortening the release cycle and LTS style releases
>> and
>>> they feel like the most direct approach.  I'm a little wary of more
>> branches
>>> since that could backfire and make the problem worse.
>> 
>> 
>> 
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, http://www.datastax.com
>> @spyced
>> 

Re: Proposed changes to C* Release Schedule

[Jonathan Ellis <jb...@gmail.com>]

I think you and Marcus are right: the tension is that while some users
want new features faster, others want to keep their cluster as stable
as possible.  Right now we kind of have the worst of both worlds -- a
release cycle slow enough that there's a long wait for new features to
stabilize, but fast enough that enterprise types are having to upgrade
more often than they're comfortable with.

What if we tried a quicker release cycle, BUT we would guarantee that
you could do a rolling upgrade until we bump the supermajor version?
So 2.0 could upgrade to 3.0 without having to go through 2.1.  (But to
go to 3.1 or 4.0 you would have to go through 3.0.)

Whether this is reasonable depends on how fast we can stabilize
releases.  2.1.0 will be a good test of this.

On Tue, Jun 24, 2014 at 8:27 AM, Chris Burroughs
<ch...@gmail.com> wrote:
> On 06/17/2014 01:16 PM, Michael Kjellman wrote:
>>
>> That being said I don’t think i’m alone by identifying the problem.
>
>
> FWIW I'm not doing anything wildly unusual and I've been on a fork for as
> long as I've been on 1.2 (and various times before).  Almost everyone being
> on 1.2 with 3 other equally weighted active branches seems like an obvious
> not-great situation for running cassandra or developing.
>
> I like the idea of shortening the release cycle and LTS style releases and
> they feel like the most direct approach.  I'm a little wary of more branches
> since that could backfire and make the problem worse.



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: Proposed changes to C* Release Schedule

[Chris Burroughs <ch...@gmail.com>]

On 06/17/2014 01:16 PM, Michael Kjellman wrote:
> That being said I don’t think i’m alone by identifying the problem.

FWIW I'm not doing anything wildly unusual and I've been on a fork for 
as long as I've been on 1.2 (and various times before).  Almost everyone 
being on 1.2 with 3 other equally weighted active branches seems like an 
obvious not-great situation for running cassandra or developing.

I like the idea of shortening the release cycle and LTS style releases 
and they feel like the most direct approach.  I'm a little wary of more 
branches since that could backfire and make the problem worse.

Re: Proposed changes to C* Release Schedule

[Marcus Eriksson <kr...@gmail.com>]

I totally understand where you are coming from, I've been in the same
situation before, but;

In my experience the only time you want new features in your database is
during development, once the application you built is in production and
stable you really _never_ want to upgrade the db until there is something
major solved (like repair in 2.1 or streaming in 2.0).

Then, even if we did backport features to these experimental branches, you
are extremely likely to fall far behind in those as well, making it equally
painful to qa/upgrade.

I think I would have appreciated having an LTS-like-version back when I was
running clusters where I could basically automate rolling minor versions
without having to worry about something breaking and then once every 6
months/year bite the bullet and do the QA needed for a major upgrade.

So maybe it would be worth it having branches/releases something like:
3.0: bug fixes
3.1: bug fixes from 3.0 + small new non-breaking features
trunk (future 4.0): everything from 3.1 + new breaking features

Of course, as others have noted, this degenerates into merging hell once
4.0 is released

/Marcus




On Tue, Jun 17, 2014 at 7:32 PM, Michael Kjellman <
mkjellman@internalcircle.com> wrote:

> Agreed. But I think shrinking the release cycle naturally will get
> features into usable releases more quickly solving b as well. :)
>
> > On Jun 17, 2014, at 10:28 AM, "Jake Luciani" <ja...@gmail.com> wrote:
> >
> > So there are two issues this proposal is trying to address:
> >
> > 1. Shrink the release cycle.
> >
> > 2. Backport things to stable releases.
> >
> > We should discuss these separately since together it's hard to discuss.
> > 1. is less controversial I would think :)
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 17, 2014 at 1:16 PM, Michael Kjellman <
> > mkjellman@internalcircle.com> wrote:
> >
> >> Totally agree — "Also — i’m aware that any additional branches/releases
> >> will add additional work for any developer that works on C*. It would be
> >> great if we could strike a balance that hopefully doesn’t add
> significant
> >> additional merging/rebasing/work for the team…”
> >>
> >> That being said I don’t think i’m alone by identifying the problem. The
> >> proposed solution was what we came up with in the hour or so we
> discussed
> >> this in person. How else can you shrink the release schedule without
> >> creating another branch? Also — the idea is to only have this branch
> >> “active” during the overlap when major release branches need to
> stabilize.
> >>
> >>> On Jun 17, 2014, at 10:03 AM, Brandon Williams <dr...@gmail.com>
> wrote:
> >>>
> >>> If that's what we want, merging is going to be much more painful.
> >>> Currently we merge:
> >>>
> >>> 1.2->2.0->2.1->3.0
> >>>
> >>> If we add an experimental branch for each, we still have to merge the
> >>> stable branch into experiemental:
> >>>
> >>> 1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex
> >>>
> >>> And then the experimentals into each other:
> >>>
> >>> 1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex
> >>>
> >>> That's quite a lot of merging in the end.
> >>>
> >>>
> >>>> On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com>
> wrote:
> >>>>
> >>>> I'm not sure many people have the problem you are describing.  This is
> >> more
> >>>> of a C* developer issue than a C* user issue.
> >>>>
> >>>>
> >>>> Is the below what you are describing we move to?:
> >>>>
> >>>> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
> >>>> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
> >>>>
> >>>> Specific changes would be backported based on the "less riskyness" of
> >> the
> >>>> change which you are assuming will be constant across versions?
> >>>>
> >>>> -Jake
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
> >>>> mkjellman@internalcircle.com> wrote:
> >>>>
> >>>>> It's a bit about features - but it's more an attempt to achieve the
> >> goals
> >>>>> of what might happen with a 4 week release cycle (but that itself --
> in
> >>>>> practice didn't prove to be valid/reasonable).
> >>>>>
> >>>>> If something like an executor service for performance is changed (for
> >>>>> example) it is definitely a more risky change than what would
> currently
> >>>> go
> >>>>> into 1.2 -- but most likely we would want to get patches like that
> >> into a
> >>>>> usable build.
> >>>>>
> >>>>> So I guess: a) reduce code drift between branches we run in
> production
> >> b)
> >>>>> get newer "features" into production faster where breaking changes
> >> aren't
> >>>>> required for the scope of the patch.
> >>>>>
> >>>>> Additionally - it's also a question of what release we use when we
> >>>>> identify an issue we want to work on internally. If we are on 1.2
> >> because
> >>>>> we can't yet take ALL of 2.0 - do we now need to target our work
> >> against
> >>>>> 1.2? I would rather write it against the months worth of changes that
> >>>> have
> >>>>> happened since.
> >>>>>
> >>>>> Finally, it's an attempt to make the internal forking not as common
> as
> >> it
> >>>>> might be today. As you said - this is somewhat of a common process.
> >>>>>
> >>>>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com>
> wrote
> >>>>>>
> >>>>>> Hi Michael,
> >>>>>>
> >>>>>> I didn't get to hear the in person conversation so taking a step
> back.
> >>>>>> The proposal seems to be in response to a common problem.  i.e.  I'm
> >> on
> >>>>> C*
> >>>>>> version X and I need feature Y which is only available on version Z.
> >> Is
> >>>>>> this correct?
> >>>>>>
> >>>>>> The options have been: a) upgrade to version Z or b) fork C* and
> >>>>> backport.
> >>>>>> Coming my my previous job where I ran a prod C* cluster I felt this
> >>>> and I
> >>>>>> expect many others do too.  We did have to fork and backport patches
> >> we
> >>>>>> needed and it was hard.
> >>>>>>
> >>>>>> This is specific to features and not bugs, since bugs are fixed in
> all
> >>>>>> versions affected.
> >>>>>>
> >>>>>> -Jake
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> >>>>>> mkjellman@internalcircle.com> wrote:
> >>>>>>
> >>>>>>> Hi Dev@ List—
> >>>>>>>
> >>>>>>> TL;DR:
> >>>>>>> I’d love it if we could modify the C* release cycle to include an
> >>>>>>> additional “experimental” release branch that straddles the current
> >>>>> major
> >>>>>>> releases that includes somewhat “untested” or “risky” commits that
> >>>>> normally
> >>>>>>> would only go into the next major release. Releases based from this
> >>>>> branch
> >>>>>>> wouldn’t contain any features that require breaking changes or are
> >>>>>>> considered highly “untested” or “risky” but would include the many
> >>>> other
> >>>>>>> commits that today are considered too unsafe to put into the
> previous
> >>>>>>> stable branch. This will allow us to run code closer to the current
> >>>>> stable
> >>>>>>> release branch when we are unable to move fully to the new major
> >>>> release
> >>>>>>> branch. Also, during the release cycle of the next major release
> >>>> branch
> >>>>> the
> >>>>>>> project can get feedback from a subset of the total changes that
> will
> >>>>>>> ultimately make it into that final new major release. Also — i’m
> >> aware
> >>>>> that
> >>>>>>> any additional branches/releases will add additional work for any
> >>>>> developer
> >>>>>>> that works on C*. It would be great if we could strike a balance
> that
> >>>>>>> hopefully doesn’t add significant additional merging/rebasing/work
> >> for
> >>>>> the
> >>>>>>> team...
> >>>>>>>
> >>>>>>> The Longer Story:
> >>>>>>> Last week I had a conversation with a few people regarding a
> proposed
> >>>>>>> change to the current C* release schedule.
> >>>>>>>
> >>>>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
> >>>>> difficult,
> >>>>>>> it would be ideal if we could better sync our internal release
> >>>> schedule
> >>>>>>> with more recent Cassandra releases. The current cycle has resulted
> >> in
> >>>>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
> >>>> stable
> >>>>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
> >>>> potential
> >>>>> for
> >>>>>>> another out-of-band 1.2/previous stable release build. We would
> love
> >>>> to
> >>>>>>> always run the current “stable” release in production but
> >>>>>>> generally/historically it takes time and a few minor releases to
> the
> >>>>>>> current “major” branch stable to get to a state where we can accept
> >>>> for
> >>>>> use
> >>>>>>> in production. Additionally, as major releases are currently used
> to
> >>>>> make
> >>>>>>> “breaking” changes that require a more involved and risky upgrade
> >>>>> process,
> >>>>>>> it’s a much bigger deal to deploy a new major into production than
> a
> >>>>>>> release without breaking changes. (upgrade-sstables for example is
> >>>>> required
> >>>>>>> when upgrading to a new major release branch. this unavoidable step
> >>>> adds
> >>>>>>> lots of temporary load to the cluster and means deploying/upgrading
> >> to
> >>>>>>> major releases tends to be a bit more risky than between minor
> >>>> releases
> >>>>> and
> >>>>>>> a more involved/long running process). This means even though there
> >>>> are
> >>>>>>> months worth of stable hard work/awesome improvements in the
> current
> >>>>>>> “stable” major release branch (today this is 2.0), we end up with
> an
> >>>>>>> unavoidable and undesired lag in getting more recent C* changes
> >> pushed
> >>>>> into
> >>>>>>> production. This means we are unable to provide feedback on newer
> >>>>> changes
> >>>>>>> sooner to the community, stuck and unable to get even a subset of
> the
> >>>>>>> awesome changes as we can’t yet take ALL the changes from the new
> >>>> major
> >>>>>>> release branch, and finally if we find an issue in production or
> want
> >>>> to
> >>>>>>> work on new functionality it would be ideal if we can write it
> >>>> against a
> >>>>>>> release that is closer to the next major release while also
> providing
> >>>>> us a
> >>>>>>> reasonable way to get the feature deployed internally on a branch
> we
> >>>> are
> >>>>>>> running.
> >>>>>>>
> >>>>>>> Currently, the project generally tends to include all
> >>>>> risky/breaking/more
> >>>>>>> “feature” oriented tickets only into the next major release +
> trunk.
> >>>>>>> However, there is a subset of these changes that are “somewhat”
> more
> >>>>> risky
> >>>>>>> changes but pose little/less/no risk the commit with introduce a
> >>>>> regression
> >>>>>>> outside of the scope of the patch/component. Additionally, any
> >> changes
> >>>>> that
> >>>>>>> depend on other higher risk/breaking commits/changes wouldn’t be
> >>>>>>> candidates for this proposed release branch. In a perfect world we
> >>>> would
> >>>>>>> love to target a new “interim” or “experimental” train of releases
> >>>>> which is
> >>>>>>> loosely the most stable current release train but also includes a
> >>>>> subset of
> >>>>>>> changes from the next major train. (While we were discussing we
> >>>> thought
> >>>>>>> about possible parallels to the concept of a LTS (Long Term
> Support)
> >>>>>>> release cycle and what some people have dubbed the “tick-tock”
> >> release
> >>>>>>> cycle.) This might look something like 1.2 branch + all
> >>>>>>> moderately-to-“less”-risky/non-breaking commits which currently
> would
> >>>>> only
> >>>>>>> end up in a 2.0 or 2.1 release. (Off the top of my head,
> immediately
> >>>> bad
> >>>>>>> candidates for this build would be for changes to components such
> as
> >>>>>>> gossip, streaming, or any patch that changes the storage format
> etc).
> >>>>> This
> >>>>>>> would enable the project to provide builds for more
> >>>> active/risk-adverse
> >>>>>>> users looking for a reasonable way to get more features and changes
> >>>> into
> >>>>>>> production than with today’s release cycle. Additionally, this
> would
> >>>>>>> hopefully facilitate/increase quicker feedback to the project on a
> >>>>> subset
> >>>>>>> of the new major release branch and any bugs found could be
> reported
> >>>>>>> against an actual reproducible release instead of some custom build
> >>>>> with a
> >>>>>>> given number of patches from Jira or git SHAs applied/backported.
> >>>>>>>
> >>>>>>> As it will always take both time and n releases to reach a stable
> >>>> minor
> >>>>>>> release for a new major train; users could deploy this new release
> to
> >>>>> get a
> >>>>>>> subset of new features and changes with higher risk than would
> >>>>> otherwise go
> >>>>>>> into a minor release of the previous stable release train. If
> >>>>> internally we
> >>>>>>> wanted to add a new feature we could target this release while
> >> testing
> >>>>>>> internally, and hopefully given the smaller delta between this
> >>>>>>> “interim/experimental” to make it easier to re-base patches into
> the
> >>>>> next
> >>>>>>> major release train. This would help us avoid what today has
> >>>>> unfortunately
> >>>>>>> become a unavoidable large lag in getting new C* builds into
> >>>> production
> >>>>> as
> >>>>>>> while we attempt to sync our internal releases with a internally/or
> >>>>>>> community QA’ed/accepted build/release of the current “stable”
> >>>>> build/branch
> >>>>>>> (currently this is 2.0).
> >>>>>>>
> >>>>>>> To accomplish this, the commit workflow would unfortunately need
> >>>> change
> >>>>>>> where an additional process is added to determine “eligibility” or
> >>>>>>> “appropriateness” of a given commit to additionally also be
> committed
> >>>> to
> >>>>>>> the “experimental” build branch (maybe it’s as simple as leaving it
> >> up
> >>>>> to
> >>>>>>> the reviewer + author to determine the risk factor and difficulty
> in
> >>>>>>> merging the change back into the “experimental” build?). If it is
> >>>> agreed
> >>>>>>> the commit/change/patch is a good candidate for the “experimental”
> >>>>> branch,
> >>>>>>> in addition to committing the patch to the current major release
> >>>> branch,
> >>>>>>> the commit would also be merged into the new “experimental”
> release.
> >>>> If
> >>>>>>> commits make it into the “experimental” branch frequently, I would
> >>>>>>> expect/hope merging patches into the “experimental” build would be
> >>>>>>> relatively easy as the “experimental” branch should also have most
> of
> >>>>> the
> >>>>>>> changes from the major release branch sans those considered highly
> >>>>> risky or
> >>>>>>> breaking. Additionally, if internally we want to work on a new
> >> feature
> >>>>> and
> >>>>>>> test internally before submitting a patch, we could target our code
> >>>>> against
> >>>>>>> the “experimental” branch, allowing us to test our changes in
> >>>> production
> >>>>>>> without forking C* internally, writing our code against more recent
> >>>>>>> “modern” changes, and then hopefully getting that work back to the
> >>>>>>> community.
> >>>>>>>
> >>>>>>>
> >>>>>>> Hope this was clear enough and accurately summarizes the
> conversation
> >>>> a
> >>>>>>> few of us had! Looking forward to everyone’s feedback and comments.
> >>>>>>>
> >>>>>>> best,
> >>>>>>> kjellman
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> http://twitter.com/tjake
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://twitter.com/tjake
> >
> >
> > --
> > http://twitter.com/tjake
>

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

Agreed. But I think shrinking the release cycle naturally will get features into usable releases more quickly solving b as well. :)

> On Jun 17, 2014, at 10:28 AM, "Jake Luciani" <ja...@gmail.com> wrote:
> 
> So there are two issues this proposal is trying to address:
> 
> 1. Shrink the release cycle.
> 
> 2. Backport things to stable releases.
> 
> We should discuss these separately since together it's hard to discuss.
> 1. is less controversial I would think :)
> 
> 
> 
> 
> 
> 
> On Tue, Jun 17, 2014 at 1:16 PM, Michael Kjellman <
> mkjellman@internalcircle.com> wrote:
> 
>> Totally agree — "Also — i’m aware that any additional branches/releases
>> will add additional work for any developer that works on C*. It would be
>> great if we could strike a balance that hopefully doesn’t add significant
>> additional merging/rebasing/work for the team…”
>> 
>> That being said I don’t think i’m alone by identifying the problem. The
>> proposed solution was what we came up with in the hour or so we discussed
>> this in person. How else can you shrink the release schedule without
>> creating another branch? Also — the idea is to only have this branch
>> “active” during the overlap when major release branches need to stabilize.
>> 
>>> On Jun 17, 2014, at 10:03 AM, Brandon Williams <dr...@gmail.com> wrote:
>>> 
>>> If that's what we want, merging is going to be much more painful.
>>> Currently we merge:
>>> 
>>> 1.2->2.0->2.1->3.0
>>> 
>>> If we add an experimental branch for each, we still have to merge the
>>> stable branch into experiemental:
>>> 
>>> 1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex
>>> 
>>> And then the experimentals into each other:
>>> 
>>> 1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex
>>> 
>>> That's quite a lot of merging in the end.
>>> 
>>> 
>>>> On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>> 
>>>> I'm not sure many people have the problem you are describing.  This is
>> more
>>>> of a C* developer issue than a C* user issue.
>>>> 
>>>> 
>>>> Is the below what you are describing we move to?:
>>>> 
>>>> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
>>>> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
>>>> 
>>>> Specific changes would be backported based on the "less riskyness" of
>> the
>>>> change which you are assuming will be constant across versions?
>>>> 
>>>> -Jake
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
>>>> mkjellman@internalcircle.com> wrote:
>>>> 
>>>>> It's a bit about features - but it's more an attempt to achieve the
>> goals
>>>>> of what might happen with a 4 week release cycle (but that itself -- in
>>>>> practice didn't prove to be valid/reasonable).
>>>>> 
>>>>> If something like an executor service for performance is changed (for
>>>>> example) it is definitely a more risky change than what would currently
>>>> go
>>>>> into 1.2 -- but most likely we would want to get patches like that
>> into a
>>>>> usable build.
>>>>> 
>>>>> So I guess: a) reduce code drift between branches we run in production
>> b)
>>>>> get newer "features" into production faster where breaking changes
>> aren't
>>>>> required for the scope of the patch.
>>>>> 
>>>>> Additionally - it's also a question of what release we use when we
>>>>> identify an issue we want to work on internally. If we are on 1.2
>> because
>>>>> we can't yet take ALL of 2.0 - do we now need to target our work
>> against
>>>>> 1.2? I would rather write it against the months worth of changes that
>>>> have
>>>>> happened since.
>>>>> 
>>>>> Finally, it's an attempt to make the internal forking not as common as
>> it
>>>>> might be today. As you said - this is somewhat of a common process.
>>>>> 
>>>>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> I didn't get to hear the in person conversation so taking a step back.
>>>>>> The proposal seems to be in response to a common problem.  i.e.  I'm
>> on
>>>>> C*
>>>>>> version X and I need feature Y which is only available on version Z.
>> Is
>>>>>> this correct?
>>>>>> 
>>>>>> The options have been: a) upgrade to version Z or b) fork C* and
>>>>> backport.
>>>>>> Coming my my previous job where I ran a prod C* cluster I felt this
>>>> and I
>>>>>> expect many others do too.  We did have to fork and backport patches
>> we
>>>>>> needed and it was hard.
>>>>>> 
>>>>>> This is specific to features and not bugs, since bugs are fixed in all
>>>>>> versions affected.
>>>>>> 
>>>>>> -Jake
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
>>>>>> mkjellman@internalcircle.com> wrote:
>>>>>> 
>>>>>>> Hi Dev@ List—
>>>>>>> 
>>>>>>> TL;DR:
>>>>>>> I’d love it if we could modify the C* release cycle to include an
>>>>>>> additional “experimental” release branch that straddles the current
>>>>> major
>>>>>>> releases that includes somewhat “untested” or “risky” commits that
>>>>> normally
>>>>>>> would only go into the next major release. Releases based from this
>>>>> branch
>>>>>>> wouldn’t contain any features that require breaking changes or are
>>>>>>> considered highly “untested” or “risky” but would include the many
>>>> other
>>>>>>> commits that today are considered too unsafe to put into the previous
>>>>>>> stable branch. This will allow us to run code closer to the current
>>>>> stable
>>>>>>> release branch when we are unable to move fully to the new major
>>>> release
>>>>>>> branch. Also, during the release cycle of the next major release
>>>> branch
>>>>> the
>>>>>>> project can get feedback from a subset of the total changes that will
>>>>>>> ultimately make it into that final new major release. Also — i’m
>> aware
>>>>> that
>>>>>>> any additional branches/releases will add additional work for any
>>>>> developer
>>>>>>> that works on C*. It would be great if we could strike a balance that
>>>>>>> hopefully doesn’t add significant additional merging/rebasing/work
>> for
>>>>> the
>>>>>>> team...
>>>>>>> 
>>>>>>> The Longer Story:
>>>>>>> Last week I had a conversation with a few people regarding a proposed
>>>>>>> change to the current C* release schedule.
>>>>>>> 
>>>>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
>>>>> difficult,
>>>>>>> it would be ideal if we could better sync our internal release
>>>> schedule
>>>>>>> with more recent Cassandra releases. The current cycle has resulted
>> in
>>>>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
>>>> stable
>>>>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
>>>> potential
>>>>> for
>>>>>>> another out-of-band 1.2/previous stable release build. We would love
>>>> to
>>>>>>> always run the current “stable” release in production but
>>>>>>> generally/historically it takes time and a few minor releases to the
>>>>>>> current “major” branch stable to get to a state where we can accept
>>>> for
>>>>> use
>>>>>>> in production. Additionally, as major releases are currently used to
>>>>> make
>>>>>>> “breaking” changes that require a more involved and risky upgrade
>>>>> process,
>>>>>>> it’s a much bigger deal to deploy a new major into production than a
>>>>>>> release without breaking changes. (upgrade-sstables for example is
>>>>> required
>>>>>>> when upgrading to a new major release branch. this unavoidable step
>>>> adds
>>>>>>> lots of temporary load to the cluster and means deploying/upgrading
>> to
>>>>>>> major releases tends to be a bit more risky than between minor
>>>> releases
>>>>> and
>>>>>>> a more involved/long running process). This means even though there
>>>> are
>>>>>>> months worth of stable hard work/awesome improvements in the current
>>>>>>> “stable” major release branch (today this is 2.0), we end up with an
>>>>>>> unavoidable and undesired lag in getting more recent C* changes
>> pushed
>>>>> into
>>>>>>> production. This means we are unable to provide feedback on newer
>>>>> changes
>>>>>>> sooner to the community, stuck and unable to get even a subset of the
>>>>>>> awesome changes as we can’t yet take ALL the changes from the new
>>>> major
>>>>>>> release branch, and finally if we find an issue in production or want
>>>> to
>>>>>>> work on new functionality it would be ideal if we can write it
>>>> against a
>>>>>>> release that is closer to the next major release while also providing
>>>>> us a
>>>>>>> reasonable way to get the feature deployed internally on a branch we
>>>> are
>>>>>>> running.
>>>>>>> 
>>>>>>> Currently, the project generally tends to include all
>>>>> risky/breaking/more
>>>>>>> “feature” oriented tickets only into the next major release + trunk.
>>>>>>> However, there is a subset of these changes that are “somewhat” more
>>>>> risky
>>>>>>> changes but pose little/less/no risk the commit with introduce a
>>>>> regression
>>>>>>> outside of the scope of the patch/component. Additionally, any
>> changes
>>>>> that
>>>>>>> depend on other higher risk/breaking commits/changes wouldn’t be
>>>>>>> candidates for this proposed release branch. In a perfect world we
>>>> would
>>>>>>> love to target a new “interim” or “experimental” train of releases
>>>>> which is
>>>>>>> loosely the most stable current release train but also includes a
>>>>> subset of
>>>>>>> changes from the next major train. (While we were discussing we
>>>> thought
>>>>>>> about possible parallels to the concept of a LTS (Long Term Support)
>>>>>>> release cycle and what some people have dubbed the “tick-tock”
>> release
>>>>>>> cycle.) This might look something like 1.2 branch + all
>>>>>>> moderately-to-“less”-risky/non-breaking commits which currently would
>>>>> only
>>>>>>> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately
>>>> bad
>>>>>>> candidates for this build would be for changes to components such as
>>>>>>> gossip, streaming, or any patch that changes the storage format etc).
>>>>> This
>>>>>>> would enable the project to provide builds for more
>>>> active/risk-adverse
>>>>>>> users looking for a reasonable way to get more features and changes
>>>> into
>>>>>>> production than with today’s release cycle. Additionally, this would
>>>>>>> hopefully facilitate/increase quicker feedback to the project on a
>>>>> subset
>>>>>>> of the new major release branch and any bugs found could be reported
>>>>>>> against an actual reproducible release instead of some custom build
>>>>> with a
>>>>>>> given number of patches from Jira or git SHAs applied/backported.
>>>>>>> 
>>>>>>> As it will always take both time and n releases to reach a stable
>>>> minor
>>>>>>> release for a new major train; users could deploy this new release to
>>>>> get a
>>>>>>> subset of new features and changes with higher risk than would
>>>>> otherwise go
>>>>>>> into a minor release of the previous stable release train. If
>>>>> internally we
>>>>>>> wanted to add a new feature we could target this release while
>> testing
>>>>>>> internally, and hopefully given the smaller delta between this
>>>>>>> “interim/experimental” to make it easier to re-base patches into the
>>>>> next
>>>>>>> major release train. This would help us avoid what today has
>>>>> unfortunately
>>>>>>> become a unavoidable large lag in getting new C* builds into
>>>> production
>>>>> as
>>>>>>> while we attempt to sync our internal releases with a internally/or
>>>>>>> community QA’ed/accepted build/release of the current “stable”
>>>>> build/branch
>>>>>>> (currently this is 2.0).
>>>>>>> 
>>>>>>> To accomplish this, the commit workflow would unfortunately need
>>>> change
>>>>>>> where an additional process is added to determine “eligibility” or
>>>>>>> “appropriateness” of a given commit to additionally also be committed
>>>> to
>>>>>>> the “experimental” build branch (maybe it’s as simple as leaving it
>> up
>>>>> to
>>>>>>> the reviewer + author to determine the risk factor and difficulty in
>>>>>>> merging the change back into the “experimental” build?). If it is
>>>> agreed
>>>>>>> the commit/change/patch is a good candidate for the “experimental”
>>>>> branch,
>>>>>>> in addition to committing the patch to the current major release
>>>> branch,
>>>>>>> the commit would also be merged into the new “experimental” release.
>>>> If
>>>>>>> commits make it into the “experimental” branch frequently, I would
>>>>>>> expect/hope merging patches into the “experimental” build would be
>>>>>>> relatively easy as the “experimental” branch should also have most of
>>>>> the
>>>>>>> changes from the major release branch sans those considered highly
>>>>> risky or
>>>>>>> breaking. Additionally, if internally we want to work on a new
>> feature
>>>>> and
>>>>>>> test internally before submitting a patch, we could target our code
>>>>> against
>>>>>>> the “experimental” branch, allowing us to test our changes in
>>>> production
>>>>>>> without forking C* internally, writing our code against more recent
>>>>>>> “modern” changes, and then hopefully getting that work back to the
>>>>>>> community.
>>>>>>> 
>>>>>>> 
>>>>>>> Hope this was clear enough and accurately summarizes the conversation
>>>> a
>>>>>>> few of us had! Looking forward to everyone’s feedback and comments.
>>>>>>> 
>>>>>>> best,
>>>>>>> kjellman
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> http://twitter.com/tjake
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://twitter.com/tjake
> 
> 
> -- 
> http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

Totally agree — "Also — i’m aware that any additional branches/releases will add additional work for any developer that works on C*. It would be great if we could strike a balance that hopefully doesn’t add significant additional merging/rebasing/work for the team…”

That being said I don’t think i’m alone by identifying the problem. The proposed solution was what we came up with in the hour or so we discussed this in person. How else can you shrink the release schedule without creating another branch? Also — the idea is to only have this branch “active” during the overlap when major release branches need to stabilize.

> On Jun 17, 2014, at 10:03 AM, Brandon Williams <dr...@gmail.com> wrote:
> 
> If that's what we want, merging is going to be much more painful.
> Currently we merge:
> 
> 1.2->2.0->2.1->3.0
> 
> If we add an experimental branch for each, we still have to merge the
> stable branch into experiemental:
> 
> 1-2->1.2ex, 2.0->2.0ex, 2.1->2.1ex, 3.0->3.0ex
> 
> And then the experimentals into each other:
> 
> 1.2ex->2.0ex, 2.0ex->2.1ex, 2.1ex->3.0ex
> 
> That's quite a lot of merging in the end.
> 
> 
> On Tue, Jun 17, 2014 at 11:51 AM, Jake Luciani <ja...@gmail.com> wrote:
> 
>> I'm not sure many people have the problem you are describing.  This is more
>> of a C* developer issue than a C* user issue.
>> 
>> 
>> Is the below what you are describing we move to?:
>> 
>> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
>> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
>> 
>> Specific changes would be backported based on the "less riskyness" of the
>> change which you are assuming will be constant across versions?
>> 
>> -Jake
>> 
>> 
>> 
>> 
>> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
>> mkjellman@internalcircle.com> wrote:
>> 
>>> It's a bit about features - but it's more an attempt to achieve the goals
>>> of what might happen with a 4 week release cycle (but that itself -- in
>>> practice didn't prove to be valid/reasonable).
>>> 
>>> If something like an executor service for performance is changed (for
>>> example) it is definitely a more risky change than what would currently
>> go
>>> into 1.2 -- but most likely we would want to get patches like that into a
>>> usable build.
>>> 
>>> So I guess: a) reduce code drift between branches we run in production b)
>>> get newer "features" into production faster where breaking changes aren't
>>> required for the scope of the patch.
>>> 
>>> Additionally - it's also a question of what release we use when we
>>> identify an issue we want to work on internally. If we are on 1.2 because
>>> we can't yet take ALL of 2.0 - do we now need to target our work against
>>> 1.2? I would rather write it against the months worth of changes that
>> have
>>> happened since.
>>> 
>>> Finally, it's an attempt to make the internal forking not as common as it
>>> might be today. As you said - this is somewhat of a common process.
>>> 
>>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
>>>> 
>>>> Hi Michael,
>>>> 
>>>> I didn't get to hear the in person conversation so taking a step back.
>>>> The proposal seems to be in response to a common problem.  i.e.  I'm on
>>> C*
>>>> version X and I need feature Y which is only available on version Z. Is
>>>> this correct?
>>>> 
>>>> The options have been: a) upgrade to version Z or b) fork C* and
>>> backport.
>>>> Coming my my previous job where I ran a prod C* cluster I felt this
>> and I
>>>> expect many others do too.  We did have to fork and backport patches we
>>>> needed and it was hard.
>>>> 
>>>> This is specific to features and not bugs, since bugs are fixed in all
>>>> versions affected.
>>>> 
>>>> -Jake
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
>>>> mkjellman@internalcircle.com> wrote:
>>>> 
>>>>> Hi Dev@ List—
>>>>> 
>>>>> TL;DR:
>>>>> I’d love it if we could modify the C* release cycle to include an
>>>>> additional “experimental” release branch that straddles the current
>>> major
>>>>> releases that includes somewhat “untested” or “risky” commits that
>>> normally
>>>>> would only go into the next major release. Releases based from this
>>> branch
>>>>> wouldn’t contain any features that require breaking changes or are
>>>>> considered highly “untested” or “risky” but would include the many
>> other
>>>>> commits that today are considered too unsafe to put into the previous
>>>>> stable branch. This will allow us to run code closer to the current
>>> stable
>>>>> release branch when we are unable to move fully to the new major
>> release
>>>>> branch. Also, during the release cycle of the next major release
>> branch
>>> the
>>>>> project can get feedback from a subset of the total changes that will
>>>>> ultimately make it into that final new major release. Also — i’m aware
>>> that
>>>>> any additional branches/releases will add additional work for any
>>> developer
>>>>> that works on C*. It would be great if we could strike a balance that
>>>>> hopefully doesn’t add significant additional merging/rebasing/work for
>>> the
>>>>> team...
>>>>> 
>>>>> The Longer Story:
>>>>> Last week I had a conversation with a few people regarding a proposed
>>>>> change to the current C* release schedule.
>>>>> 
>>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
>>> difficult,
>>>>> it would be ideal if we could better sync our internal release
>> schedule
>>>>> with more recent Cassandra releases. The current cycle has resulted in
>>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official
>> stable
>>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the
>> potential
>>> for
>>>>> another out-of-band 1.2/previous stable release build. We would love
>> to
>>>>> always run the current “stable” release in production but
>>>>> generally/historically it takes time and a few minor releases to the
>>>>> current “major” branch stable to get to a state where we can accept
>> for
>>> use
>>>>> in production. Additionally, as major releases are currently used to
>>> make
>>>>> “breaking” changes that require a more involved and risky upgrade
>>> process,
>>>>> it’s a much bigger deal to deploy a new major into production than a
>>>>> release without breaking changes. (upgrade-sstables for example is
>>> required
>>>>> when upgrading to a new major release branch. this unavoidable step
>> adds
>>>>> lots of temporary load to the cluster and means deploying/upgrading to
>>>>> major releases tends to be a bit more risky than between minor
>> releases
>>> and
>>>>> a more involved/long running process). This means even though there
>> are
>>>>> months worth of stable hard work/awesome improvements in the current
>>>>> “stable” major release branch (today this is 2.0), we end up with an
>>>>> unavoidable and undesired lag in getting more recent C* changes pushed
>>> into
>>>>> production. This means we are unable to provide feedback on newer
>>> changes
>>>>> sooner to the community, stuck and unable to get even a subset of the
>>>>> awesome changes as we can’t yet take ALL the changes from the new
>> major
>>>>> release branch, and finally if we find an issue in production or want
>> to
>>>>> work on new functionality it would be ideal if we can write it
>> against a
>>>>> release that is closer to the next major release while also providing
>>> us a
>>>>> reasonable way to get the feature deployed internally on a branch we
>> are
>>>>> running.
>>>>> 
>>>>> Currently, the project generally tends to include all
>>> risky/breaking/more
>>>>> “feature” oriented tickets only into the next major release + trunk.
>>>>> However, there is a subset of these changes that are “somewhat” more
>>> risky
>>>>> changes but pose little/less/no risk the commit with introduce a
>>> regression
>>>>> outside of the scope of the patch/component. Additionally, any changes
>>> that
>>>>> depend on other higher risk/breaking commits/changes wouldn’t be
>>>>> candidates for this proposed release branch. In a perfect world we
>> would
>>>>> love to target a new “interim” or “experimental” train of releases
>>> which is
>>>>> loosely the most stable current release train but also includes a
>>> subset of
>>>>> changes from the next major train. (While we were discussing we
>> thought
>>>>> about possible parallels to the concept of a LTS (Long Term Support)
>>>>> release cycle and what some people have dubbed the “tick-tock” release
>>>>> cycle.) This might look something like 1.2 branch + all
>>>>> moderately-to-“less”-risky/non-breaking commits which currently would
>>> only
>>>>> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately
>> bad
>>>>> candidates for this build would be for changes to components such as
>>>>> gossip, streaming, or any patch that changes the storage format etc).
>>> This
>>>>> would enable the project to provide builds for more
>> active/risk-adverse
>>>>> users looking for a reasonable way to get more features and changes
>> into
>>>>> production than with today’s release cycle. Additionally, this would
>>>>> hopefully facilitate/increase quicker feedback to the project on a
>>> subset
>>>>> of the new major release branch and any bugs found could be reported
>>>>> against an actual reproducible release instead of some custom build
>>> with a
>>>>> given number of patches from Jira or git SHAs applied/backported.
>>>>> 
>>>>> As it will always take both time and n releases to reach a stable
>> minor
>>>>> release for a new major train; users could deploy this new release to
>>> get a
>>>>> subset of new features and changes with higher risk than would
>>> otherwise go
>>>>> into a minor release of the previous stable release train. If
>>> internally we
>>>>> wanted to add a new feature we could target this release while testing
>>>>> internally, and hopefully given the smaller delta between this
>>>>> “interim/experimental” to make it easier to re-base patches into the
>>> next
>>>>> major release train. This would help us avoid what today has
>>> unfortunately
>>>>> become a unavoidable large lag in getting new C* builds into
>> production
>>> as
>>>>> while we attempt to sync our internal releases with a internally/or
>>>>> community QA’ed/accepted build/release of the current “stable”
>>> build/branch
>>>>> (currently this is 2.0).
>>>>> 
>>>>> To accomplish this, the commit workflow would unfortunately need
>> change
>>>>> where an additional process is added to determine “eligibility” or
>>>>> “appropriateness” of a given commit to additionally also be committed
>> to
>>>>> the “experimental” build branch (maybe it’s as simple as leaving it up
>>> to
>>>>> the reviewer + author to determine the risk factor and difficulty in
>>>>> merging the change back into the “experimental” build?). If it is
>> agreed
>>>>> the commit/change/patch is a good candidate for the “experimental”
>>> branch,
>>>>> in addition to committing the patch to the current major release
>> branch,
>>>>> the commit would also be merged into the new “experimental” release.
>> If
>>>>> commits make it into the “experimental” branch frequently, I would
>>>>> expect/hope merging patches into the “experimental” build would be
>>>>> relatively easy as the “experimental” branch should also have most of
>>> the
>>>>> changes from the major release branch sans those considered highly
>>> risky or
>>>>> breaking. Additionally, if internally we want to work on a new feature
>>> and
>>>>> test internally before submitting a patch, we could target our code
>>> against
>>>>> the “experimental” branch, allowing us to test our changes in
>> production
>>>>> without forking C* internally, writing our code against more recent
>>>>> “modern” changes, and then hopefully getting that work back to the
>>>>> community.
>>>>> 
>>>>> 
>>>>> Hope this was clear enough and accurately summarizes the conversation
>> a
>>>>> few of us had! Looking forward to everyone’s feedback and comments.
>>>>> 
>>>>> best,
>>>>> kjellman
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> http://twitter.com/tjake
>>> 
>> 
>> 
>> 
>> --
>> http://twitter.com/tjake
>> 

Re: Proposed changes to C* Release Schedule

[Jake Luciani <ja...@gmail.com>]

I'm not sure many people have the problem you are describing.  This is more
of a C* developer issue than a C* user issue.


Is the below what you are describing we move to?:

1.2 -> 2.0 -> 2.1 -> 3.0 stable
1.2 <- 2.0 <- 2.1 <- 3.0 experimental

Specific changes would be backported based on the "less riskyness" of the
change which you are assuming will be constant across versions?

-Jake




On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
mkjellman@internalcircle.com> wrote:

> It's a bit about features - but it's more an attempt to achieve the goals
> of what might happen with a 4 week release cycle (but that itself -- in
> practice didn't prove to be valid/reasonable).
>
> If something like an executor service for performance is changed (for
> example) it is definitely a more risky change than what would currently go
> into 1.2 -- but most likely we would want to get patches like that into a
> usable build.
>
> So I guess: a) reduce code drift between branches we run in production b)
> get newer "features" into production faster where breaking changes aren't
> required for the scope of the patch.
>
> Additionally - it's also a question of what release we use when we
> identify an issue we want to work on internally. If we are on 1.2 because
> we can't yet take ALL of 2.0 - do we now need to target our work against
> 1.2? I would rather write it against the months worth of changes that have
> happened since.
>
> Finally, it's an attempt to make the internal forking not as common as it
> might be today. As you said - this is somewhat of a common process.
>
> > On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
> >
> > Hi Michael,
> >
> > I didn't get to hear the in person conversation so taking a step back.
> > The proposal seems to be in response to a common problem.  i.e.  I'm on
> C*
> > version X and I need feature Y which is only available on version Z. Is
> > this correct?
> >
> > The options have been: a) upgrade to version Z or b) fork C* and
> backport.
> > Coming my my previous job where I ran a prod C* cluster I felt this and I
> > expect many others do too.  We did have to fork and backport patches we
> > needed and it was hard.
> >
> > This is specific to features and not bugs, since bugs are fixed in all
> > versions affected.
> >
> > -Jake
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> > mkjellman@internalcircle.com> wrote:
> >
> >> Hi Dev@ List—
> >>
> >> TL;DR:
> >> I’d love it if we could modify the C* release cycle to include an
> >> additional “experimental” release branch that straddles the current
> major
> >> releases that includes somewhat “untested” or “risky” commits that
> normally
> >> would only go into the next major release. Releases based from this
> branch
> >> wouldn’t contain any features that require breaking changes or are
> >> considered highly “untested” or “risky” but would include the many other
> >> commits that today are considered too unsafe to put into the previous
> >> stable branch. This will allow us to run code closer to the current
> stable
> >> release branch when we are unable to move fully to the new major release
> >> branch. Also, during the release cycle of the next major release branch
> the
> >> project can get feedback from a subset of the total changes that will
> >> ultimately make it into that final new major release. Also — i’m aware
> that
> >> any additional branches/releases will add additional work for any
> developer
> >> that works on C*. It would be great if we could strike a balance that
> >> hopefully doesn’t add significant additional merging/rebasing/work for
> the
> >> team...
> >>
> >> The Longer Story:
> >> Last week I had a conversation with a few people regarding a proposed
> >> change to the current C* release schedule.
> >>
> >> Other than an attempt to make Jonathan and Sylvian’s lives more
> difficult,
> >> it would be ideal if we could better sync our internal release schedule
> >> with more recent Cassandra releases. The current cycle has resulted in
> >> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable
> >> releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential
> for
> >> another out-of-band 1.2/previous stable release build. We would love to
> >> always run the current “stable” release in production but
> >> generally/historically it takes time and a few minor releases to the
> >> current “major” branch stable to get to a state where we can accept for
> use
> >> in production. Additionally, as major releases are currently used to
> make
> >> “breaking” changes that require a more involved and risky upgrade
> process,
> >> it’s a much bigger deal to deploy a new major into production than a
> >> release without breaking changes. (upgrade-sstables for example is
> required
> >> when upgrading to a new major release branch. this unavoidable step adds
> >> lots of temporary load to the cluster and means deploying/upgrading to
> >> major releases tends to be a bit more risky than between minor releases
> and
> >> a more involved/long running process). This means even though there are
> >> months worth of stable hard work/awesome improvements in the current
> >> “stable” major release branch (today this is 2.0), we end up with an
> >> unavoidable and undesired lag in getting more recent C* changes pushed
> into
> >> production. This means we are unable to provide feedback on newer
> changes
> >> sooner to the community, stuck and unable to get even a subset of the
> >> awesome changes as we can’t yet take ALL the changes from the new major
> >> release branch, and finally if we find an issue in production or want to
> >> work on new functionality it would be ideal if we can write it against a
> >> release that is closer to the next major release while also providing
> us a
> >> reasonable way to get the feature deployed internally on a branch we are
> >> running.
> >>
> >> Currently, the project generally tends to include all
> risky/breaking/more
> >> “feature” oriented tickets only into the next major release + trunk.
> >> However, there is a subset of these changes that are “somewhat” more
> risky
> >> changes but pose little/less/no risk the commit with introduce a
> regression
> >> outside of the scope of the patch/component. Additionally, any changes
> that
> >> depend on other higher risk/breaking commits/changes wouldn’t be
> >> candidates for this proposed release branch. In a perfect world we would
> >> love to target a new “interim” or “experimental” train of releases
> which is
> >> loosely the most stable current release train but also includes a
> subset of
> >> changes from the next major train. (While we were discussing we thought
> >> about possible parallels to the concept of a LTS (Long Term Support)
> >> release cycle and what some people have dubbed the “tick-tock” release
> >> cycle.) This might look something like 1.2 branch + all
> >> moderately-to-“less”-risky/non-breaking commits which currently would
> only
> >> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad
> >> candidates for this build would be for changes to components such as
> >> gossip, streaming, or any patch that changes the storage format etc).
> This
> >> would enable the project to provide builds for more active/risk-adverse
> >> users looking for a reasonable way to get more features and changes into
> >> production than with today’s release cycle. Additionally, this would
> >> hopefully facilitate/increase quicker feedback to the project on a
> subset
> >> of the new major release branch and any bugs found could be reported
> >> against an actual reproducible release instead of some custom build
> with a
> >> given number of patches from Jira or git SHAs applied/backported.
> >>
> >> As it will always take both time and n releases to reach a stable minor
> >> release for a new major train; users could deploy this new release to
> get a
> >> subset of new features and changes with higher risk than would
> otherwise go
> >> into a minor release of the previous stable release train. If
> internally we
> >> wanted to add a new feature we could target this release while testing
> >> internally, and hopefully given the smaller delta between this
> >> “interim/experimental” to make it easier to re-base patches into the
> next
> >> major release train. This would help us avoid what today has
> unfortunately
> >> become a unavoidable large lag in getting new C* builds into production
> as
> >> while we attempt to sync our internal releases with a internally/or
> >> community QA’ed/accepted build/release of the current “stable”
> build/branch
> >> (currently this is 2.0).
> >>
> >> To accomplish this, the commit workflow would unfortunately need change
> >> where an additional process is added to determine “eligibility” or
> >> “appropriateness” of a given commit to additionally also be committed to
> >> the “experimental” build branch (maybe it’s as simple as leaving it up
> to
> >> the reviewer + author to determine the risk factor and difficulty in
> >> merging the change back into the “experimental” build?). If it is agreed
> >> the commit/change/patch is a good candidate for the “experimental”
> branch,
> >> in addition to committing the patch to the current major release branch,
> >> the commit would also be merged into the new “experimental” release. If
> >> commits make it into the “experimental” branch frequently, I would
> >> expect/hope merging patches into the “experimental” build would be
> >> relatively easy as the “experimental” branch should also have most of
> the
> >> changes from the major release branch sans those considered highly
> risky or
> >> breaking. Additionally, if internally we want to work on a new feature
> and
> >> test internally before submitting a patch, we could target our code
> against
> >> the “experimental” branch, allowing us to test our changes in production
> >> without forking C* internally, writing our code against more recent
> >> “modern” changes, and then hopefully getting that work back to the
> >> community.
> >>
> >>
> >> Hope this was clear enough and accurately summarizes the conversation a
> >> few of us had! Looking forward to everyone’s feedback and comments.
> >>
> >> best,
> >> kjellman
> >
> >
> >
> >
> > --
> > http://twitter.com/tjake
>



-- 
http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

I agree — but there are lots of people though who lag in adoption while waiting for more “risky” or “breaking” changes to shake out/stabilize that want to continue deploying newer fixes. No?

> On Jun 17, 2014, at 9:51 AM, Jake Luciani <ja...@gmail.com> wrote:
> 
> I'm not sure many people have the problem you are describing.  This is more
> of a C* developer issue than a C* user issue.
> 
> 
> Is the below what you are describing we move to?:
> 
> 1.2 -> 2.0 -> 2.1 -> 3.0 stable
> 1.2 <- 2.0 <- 2.1 <- 3.0 experimental
> 
> Specific changes would be backported based on the "less riskyness" of the
> change which you are assuming will be constant across versions?
> 
> -Jake
> 
> 
> 
> 
> On Tue, Jun 17, 2014 at 12:28 PM, Michael Kjellman <
> mkjellman@internalcircle.com> wrote:
> 
>> It's a bit about features - but it's more an attempt to achieve the goals
>> of what might happen with a 4 week release cycle (but that itself -- in
>> practice didn't prove to be valid/reasonable).
>> 
>> If something like an executor service for performance is changed (for
>> example) it is definitely a more risky change than what would currently go
>> into 1.2 -- but most likely we would want to get patches like that into a
>> usable build.
>> 
>> So I guess: a) reduce code drift between branches we run in production b)
>> get newer "features" into production faster where breaking changes aren't
>> required for the scope of the patch.
>> 
>> Additionally - it's also a question of what release we use when we
>> identify an issue we want to work on internally. If we are on 1.2 because
>> we can't yet take ALL of 2.0 - do we now need to target our work against
>> 1.2? I would rather write it against the months worth of changes that have
>> happened since.
>> 
>> Finally, it's an attempt to make the internal forking not as common as it
>> might be today. As you said - this is somewhat of a common process.
>> 
>>> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
>>> 
>>> Hi Michael,
>>> 
>>> I didn't get to hear the in person conversation so taking a step back.
>>> The proposal seems to be in response to a common problem.  i.e.  I'm on
>> C*
>>> version X and I need feature Y which is only available on version Z. Is
>>> this correct?
>>> 
>>> The options have been: a) upgrade to version Z or b) fork C* and
>> backport.
>>> Coming my my previous job where I ran a prod C* cluster I felt this and I
>>> expect many others do too.  We did have to fork and backport patches we
>>> needed and it was hard.
>>> 
>>> This is specific to features and not bugs, since bugs are fixed in all
>>> versions affected.
>>> 
>>> -Jake
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
>>> mkjellman@internalcircle.com> wrote:
>>> 
>>>> Hi Dev@ List—
>>>> 
>>>> TL;DR:
>>>> I’d love it if we could modify the C* release cycle to include an
>>>> additional “experimental” release branch that straddles the current
>> major
>>>> releases that includes somewhat “untested” or “risky” commits that
>> normally
>>>> would only go into the next major release. Releases based from this
>> branch
>>>> wouldn’t contain any features that require breaking changes or are
>>>> considered highly “untested” or “risky” but would include the many other
>>>> commits that today are considered too unsafe to put into the previous
>>>> stable branch. This will allow us to run code closer to the current
>> stable
>>>> release branch when we are unable to move fully to the new major release
>>>> branch. Also, during the release cycle of the next major release branch
>> the
>>>> project can get feedback from a subset of the total changes that will
>>>> ultimately make it into that final new major release. Also — i’m aware
>> that
>>>> any additional branches/releases will add additional work for any
>> developer
>>>> that works on C*. It would be great if we could strike a balance that
>>>> hopefully doesn’t add significant additional merging/rebasing/work for
>> the
>>>> team...
>>>> 
>>>> The Longer Story:
>>>> Last week I had a conversation with a few people regarding a proposed
>>>> change to the current C* release schedule.
>>>> 
>>>> Other than an attempt to make Jonathan and Sylvian’s lives more
>> difficult,
>>>> it would be ideal if we could better sync our internal release schedule
>>>> with more recent Cassandra releases. The current cycle has resulted in
>>>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable
>>>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential
>> for
>>>> another out-of-band 1.2/previous stable release build. We would love to
>>>> always run the current “stable” release in production but
>>>> generally/historically it takes time and a few minor releases to the
>>>> current “major” branch stable to get to a state where we can accept for
>> use
>>>> in production. Additionally, as major releases are currently used to
>> make
>>>> “breaking” changes that require a more involved and risky upgrade
>> process,
>>>> it’s a much bigger deal to deploy a new major into production than a
>>>> release without breaking changes. (upgrade-sstables for example is
>> required
>>>> when upgrading to a new major release branch. this unavoidable step adds
>>>> lots of temporary load to the cluster and means deploying/upgrading to
>>>> major releases tends to be a bit more risky than between minor releases
>> and
>>>> a more involved/long running process). This means even though there are
>>>> months worth of stable hard work/awesome improvements in the current
>>>> “stable” major release branch (today this is 2.0), we end up with an
>>>> unavoidable and undesired lag in getting more recent C* changes pushed
>> into
>>>> production. This means we are unable to provide feedback on newer
>> changes
>>>> sooner to the community, stuck and unable to get even a subset of the
>>>> awesome changes as we can’t yet take ALL the changes from the new major
>>>> release branch, and finally if we find an issue in production or want to
>>>> work on new functionality it would be ideal if we can write it against a
>>>> release that is closer to the next major release while also providing
>> us a
>>>> reasonable way to get the feature deployed internally on a branch we are
>>>> running.
>>>> 
>>>> Currently, the project generally tends to include all
>> risky/breaking/more
>>>> “feature” oriented tickets only into the next major release + trunk.
>>>> However, there is a subset of these changes that are “somewhat” more
>> risky
>>>> changes but pose little/less/no risk the commit with introduce a
>> regression
>>>> outside of the scope of the patch/component. Additionally, any changes
>> that
>>>> depend on other higher risk/breaking commits/changes wouldn’t be
>>>> candidates for this proposed release branch. In a perfect world we would
>>>> love to target a new “interim” or “experimental” train of releases
>> which is
>>>> loosely the most stable current release train but also includes a
>> subset of
>>>> changes from the next major train. (While we were discussing we thought
>>>> about possible parallels to the concept of a LTS (Long Term Support)
>>>> release cycle and what some people have dubbed the “tick-tock” release
>>>> cycle.) This might look something like 1.2 branch + all
>>>> moderately-to-“less”-risky/non-breaking commits which currently would
>> only
>>>> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad
>>>> candidates for this build would be for changes to components such as
>>>> gossip, streaming, or any patch that changes the storage format etc).
>> This
>>>> would enable the project to provide builds for more active/risk-adverse
>>>> users looking for a reasonable way to get more features and changes into
>>>> production than with today’s release cycle. Additionally, this would
>>>> hopefully facilitate/increase quicker feedback to the project on a
>> subset
>>>> of the new major release branch and any bugs found could be reported
>>>> against an actual reproducible release instead of some custom build
>> with a
>>>> given number of patches from Jira or git SHAs applied/backported.
>>>> 
>>>> As it will always take both time and n releases to reach a stable minor
>>>> release for a new major train; users could deploy this new release to
>> get a
>>>> subset of new features and changes with higher risk than would
>> otherwise go
>>>> into a minor release of the previous stable release train. If
>> internally we
>>>> wanted to add a new feature we could target this release while testing
>>>> internally, and hopefully given the smaller delta between this
>>>> “interim/experimental” to make it easier to re-base patches into the
>> next
>>>> major release train. This would help us avoid what today has
>> unfortunately
>>>> become a unavoidable large lag in getting new C* builds into production
>> as
>>>> while we attempt to sync our internal releases with a internally/or
>>>> community QA’ed/accepted build/release of the current “stable”
>> build/branch
>>>> (currently this is 2.0).
>>>> 
>>>> To accomplish this, the commit workflow would unfortunately need change
>>>> where an additional process is added to determine “eligibility” or
>>>> “appropriateness” of a given commit to additionally also be committed to
>>>> the “experimental” build branch (maybe it’s as simple as leaving it up
>> to
>>>> the reviewer + author to determine the risk factor and difficulty in
>>>> merging the change back into the “experimental” build?). If it is agreed
>>>> the commit/change/patch is a good candidate for the “experimental”
>> branch,
>>>> in addition to committing the patch to the current major release branch,
>>>> the commit would also be merged into the new “experimental” release. If
>>>> commits make it into the “experimental” branch frequently, I would
>>>> expect/hope merging patches into the “experimental” build would be
>>>> relatively easy as the “experimental” branch should also have most of
>> the
>>>> changes from the major release branch sans those considered highly
>> risky or
>>>> breaking. Additionally, if internally we want to work on a new feature
>> and
>>>> test internally before submitting a patch, we could target our code
>> against
>>>> the “experimental” branch, allowing us to test our changes in production
>>>> without forking C* internally, writing our code against more recent
>>>> “modern” changes, and then hopefully getting that work back to the
>>>> community.
>>>> 
>>>> 
>>>> Hope this was clear enough and accurately summarizes the conversation a
>>>> few of us had! Looking forward to everyone’s feedback and comments.
>>>> 
>>>> best,
>>>> kjellman
>>> 
>>> 
>>> 
>>> 
>>> --
>>> http://twitter.com/tjake
>> 
> 
> 
> 
> -- 
> http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

It's a bit about features - but it's more an attempt to achieve the goals of what might happen with a 4 week release cycle (but that itself -- in practice didn't prove to be valid/reasonable). 

If something like an executor service for performance is changed (for example) it is definitely a more risky change than what would currently go into 1.2 -- but most likely we would want to get patches like that into a usable build.

So I guess: a) reduce code drift between branches we run in production b) get newer "features" into production faster where breaking changes aren't required for the scope of the patch. 

Additionally - it's also a question of what release we use when we identify an issue we want to work on internally. If we are on 1.2 because we can't yet take ALL of 2.0 - do we now need to target our work against 1.2? I would rather write it against the months worth of changes that have happened since. 

Finally, it's an attempt to make the internal forking not as common as it might be today. As you said - this is somewhat of a common process.

> On Jun 17, 2014, at 8:52 AM, "Jake Luciani" <ja...@gmail.com> wrote
> 
> Hi Michael,
> 
> I didn't get to hear the in person conversation so taking a step back.
> The proposal seems to be in response to a common problem.  i.e.  I'm on C*
> version X and I need feature Y which is only available on version Z. Is
> this correct?
> 
> The options have been: a) upgrade to version Z or b) fork C* and backport.
> Coming my my previous job where I ran a prod C* cluster I felt this and I
> expect many others do too.  We did have to fork and backport patches we
> needed and it was hard.
> 
> This is specific to features and not bugs, since bugs are fixed in all
> versions affected.
> 
> -Jake
> 
> 
> 
> 
> 
> 
> On Tue, Jun 17, 2014 at 3:16 AM, Michael Kjellman <
> mkjellman@internalcircle.com> wrote:
> 
>> Hi Dev@ List—
>> 
>> TL;DR:
>> I’d love it if we could modify the C* release cycle to include an
>> additional “experimental” release branch that straddles the current major
>> releases that includes somewhat “untested” or “risky” commits that normally
>> would only go into the next major release. Releases based from this branch
>> wouldn’t contain any features that require breaking changes or are
>> considered highly “untested” or “risky” but would include the many other
>> commits that today are considered too unsafe to put into the previous
>> stable branch. This will allow us to run code closer to the current stable
>> release branch when we are unable to move fully to the new major release
>> branch. Also, during the release cycle of the next major release branch the
>> project can get feedback from a subset of the total changes that will
>> ultimately make it into that final new major release. Also — i’m aware that
>> any additional branches/releases will add additional work for any developer
>> that works on C*. It would be great if we could strike a balance that
>> hopefully doesn’t add significant additional merging/rebasing/work for the
>> team...
>> 
>> The Longer Story:
>> Last week I had a conversation with a few people regarding a proposed
>> change to the current C* release schedule.
>> 
>> Other than an attempt to make Jonathan and Sylvian’s lives more difficult,
>> it would be ideal if we could better sync our internal release schedule
>> with more recent Cassandra releases. The current cycle has resulted in
>> currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable
>> releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential for
>> another out-of-band 1.2/previous stable release build. We would love to
>> always run the current “stable” release in production but
>> generally/historically it takes time and a few minor releases to the
>> current “major” branch stable to get to a state where we can accept for use
>> in production. Additionally, as major releases are currently used to make
>> “breaking” changes that require a more involved and risky upgrade process,
>> it’s a much bigger deal to deploy a new major into production than a
>> release without breaking changes. (upgrade-sstables for example is required
>> when upgrading to a new major release branch. this unavoidable step adds
>> lots of temporary load to the cluster and means deploying/upgrading to
>> major releases tends to be a bit more risky than between minor releases and
>> a more involved/long running process). This means even though there are
>> months worth of stable hard work/awesome improvements in the current
>> “stable” major release branch (today this is 2.0), we end up with an
>> unavoidable and undesired lag in getting more recent C* changes pushed into
>> production. This means we are unable to provide feedback on newer changes
>> sooner to the community, stuck and unable to get even a subset of the
>> awesome changes as we can’t yet take ALL the changes from the new major
>> release branch, and finally if we find an issue in production or want to
>> work on new functionality it would be ideal if we can write it against a
>> release that is closer to the next major release while also providing us a
>> reasonable way to get the feature deployed internally on a branch we are
>> running.
>> 
>> Currently, the project generally tends to include all risky/breaking/more
>> “feature” oriented tickets only into the next major release + trunk.
>> However, there is a subset of these changes that are “somewhat” more risky
>> changes but pose little/less/no risk the commit with introduce a regression
>> outside of the scope of the patch/component. Additionally, any changes that
>> depend on other higher risk/breaking commits/changes wouldn’t be
>> candidates for this proposed release branch. In a perfect world we would
>> love to target a new “interim” or “experimental” train of releases which is
>> loosely the most stable current release train but also includes a subset of
>> changes from the next major train. (While we were discussing we thought
>> about possible parallels to the concept of a LTS (Long Term Support)
>> release cycle and what some people have dubbed the “tick-tock” release
>> cycle.) This might look something like 1.2 branch + all
>> moderately-to-“less”-risky/non-breaking commits which currently would only
>> end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad
>> candidates for this build would be for changes to components such as
>> gossip, streaming, or any patch that changes the storage format etc). This
>> would enable the project to provide builds for more active/risk-adverse
>> users looking for a reasonable way to get more features and changes into
>> production than with today’s release cycle. Additionally, this would
>> hopefully facilitate/increase quicker feedback to the project on a subset
>> of the new major release branch and any bugs found could be reported
>> against an actual reproducible release instead of some custom build with a
>> given number of patches from Jira or git SHAs applied/backported.
>> 
>> As it will always take both time and n releases to reach a stable minor
>> release for a new major train; users could deploy this new release to get a
>> subset of new features and changes with higher risk than would otherwise go
>> into a minor release of the previous stable release train. If internally we
>> wanted to add a new feature we could target this release while testing
>> internally, and hopefully given the smaller delta between this
>> “interim/experimental” to make it easier to re-base patches into the next
>> major release train. This would help us avoid what today has unfortunately
>> become a unavoidable large lag in getting new C* builds into production as
>> while we attempt to sync our internal releases with a internally/or
>> community QA’ed/accepted build/release of the current “stable” build/branch
>> (currently this is 2.0).
>> 
>> To accomplish this, the commit workflow would unfortunately need change
>> where an additional process is added to determine “eligibility” or
>> “appropriateness” of a given commit to additionally also be committed to
>> the “experimental” build branch (maybe it’s as simple as leaving it up to
>> the reviewer + author to determine the risk factor and difficulty in
>> merging the change back into the “experimental” build?). If it is agreed
>> the commit/change/patch is a good candidate for the “experimental” branch,
>> in addition to committing the patch to the current major release branch,
>> the commit would also be merged into the new “experimental” release. If
>> commits make it into the “experimental” branch frequently, I would
>> expect/hope merging patches into the “experimental” build would be
>> relatively easy as the “experimental” branch should also have most of the
>> changes from the major release branch sans those considered highly risky or
>> breaking. Additionally, if internally we want to work on a new feature and
>> test internally before submitting a patch, we could target our code against
>> the “experimental” branch, allowing us to test our changes in production
>> without forking C* internally, writing our code against more recent
>> “modern” changes, and then hopefully getting that work back to the
>> community.
>> 
>> 
>> Hope this was clear enough and accurately summarizes the conversation a
>> few of us had! Looking forward to everyone’s feedback and comments.
>> 
>> best,
>> kjellman
> 
> 
> 
> 
> -- 
> http://twitter.com/tjake

Re: Proposed changes to C* Release Schedule

[Michael Kjellman <mk...@internalcircle.com>]

No, it generally takes months to reach a minor revision of the current major release to reach a release stable enough for most to use in production even if they “live on the edge". Generally there ends up being a very low number of users who've actively deployed released versions 2.0.0-2.0.5 as they too need to evaluate the build and test it in their QA/Staging environments. There are plenty of changes that went into 2.0 that are less risky that are good changes for the “I’m living on the edge” crowd sans the breaking changes and larger risky factors. (I thought i had made this clear — so I apologize I apparently didn’t). Additionally, the longer we have to deploy releases from a particular branch (while a new “stable" branches matures), the farther that branch deviates from trunk. This makes it very hard for us to target development against, especially as we do everything in our power to avoid any sort of branching or running of a special or non-released build.

best,
michael


> On Jun 17, 2014, at 12:42 AM, Jacob Rhoden <ja...@me.com> wrote:
> 
> Isn't this how it works now? Aka
> 
> 2.0 is the "I'm risk averse" stable, and
> 2.1 is the "I'm living on the edge" stable 
> 
> ______________________________
> Sent from iPhone
> 
>> On 17 Jun 2014, at 5:16 pm, Michael Kjellman <mk...@internalcircle.com> wrote:
>> 
>> Hi Dev@ List—
>> 
>> TL;DR:
>> I’d love it if we could modify the C* release cycle to include an additional “experimental” release branch that straddles the current major releases that includes somewhat “untested” or “risky” commits that normally would only go into the next major release. Releases based from this branch wouldn’t contain any features that require breaking changes or are considered highly “untested” or “risky” but would include the many other commits that today are considered too unsafe to put into the previous stable branch. This will allow us to run code closer to the current stable release branch when we are unable to move fully to the new major release branch. Also, during the release cycle of the next major release branch the project can get feedback from a subset of the total changes that will ultimately make it into that final new major release. Also — i’m aware that any additional branches/releases will add additional work for any developer that works on C*. It would be great if we could strike a balance that hopefully doesn’t add significant additional merging/rebasing/work for the team...
>> 
>> The Longer Story:
>> Last week I had a conversation with a few people regarding a proposed change to the current C* release schedule. 
>> 
>> Other than an attempt to make Jonathan and Sylvian’s lives more difficult, it would be ideal if we could better sync our internal release schedule with more recent Cassandra releases. The current cycle has resulted in currently “active” branches for 1.2, 2.0, 2.1, and +3.0. Official stable releases are from 2.0, beta’s/RC’s from 2.1, and there is the potential for another out-of-band 1.2/previous stable release build. We would love to always run the current “stable” release in production but generally/historically it takes time and a few minor releases to the current “major” branch stable to get to a state where we can accept for use in production. Additionally, as major releases are currently used to make “breaking” changes that require a more involved and risky upgrade process, it’s a much bigger deal to deploy a new major into production than a release without breaking changes. (upgrade-sstables for example is required when upgrading to a new major release branch. this unavoidable step adds lots of temporary load to the cluster and means deploying/upgrading to major releases tends to be a bit more risky than between minor releases and a more involved/long running process). This means even though there are months worth of stable hard work/awesome improvements in the current “stable” major release branch (today this is 2.0), we end up with an unavoidable and undesired lag in getting more recent C* changes pushed into production. This means we are unable to provide feedback on newer changes sooner to the community, stuck and unable to get even a subset of the awesome changes as we can’t yet take ALL the changes from the new major release branch, and finally if we find an issue in production or want to work on new functionality it would be ideal if we can write it against a release that is closer to the next major release while also providing us a reasonable way to get the feature deployed internally on a branch we are running.
>> 
>> Currently, the project generally tends to include all risky/breaking/more “feature” oriented tickets only into the next major release + trunk. However, there is a subset of these changes that are “somewhat” more risky changes but pose little/less/no risk the commit with introduce a regression outside of the scope of the patch/component. Additionally, any changes that  depend on other higher risk/breaking commits/changes wouldn’t be candidates for this proposed release branch. In a perfect world we would love to target a new “interim” or “experimental” train of releases which is loosely the most stable current release train but also includes a subset of changes from the next major train. (While we were discussing we thought about possible parallels to the concept of a LTS (Long Term Support) release cycle and what some people have dubbed the “tick-tock” release cycle.) This might look something like 1.2 branch + all moderately-to-“less”-risky/non-breaking commits which currently would only end up in a 2.0 or 2.1 release. (Off the top of my head, immediately bad candidates for this build would be for changes to components such as gossip, streaming, or any patch that changes the storage format etc). This would enable the project to provide builds for more active/risk-adverse users looking for a reasonable way to get more features and changes into production than with today’s release cycle. Additionally, this would hopefully facilitate/increase quicker feedback to the project on a subset of the new major release branch and any bugs found could be reported against an actual reproducible release instead of some custom build with a given number of patches from Jira or git SHAs applied/backported.
>> 
>> As it will always take both time and n releases to reach a stable minor release for a new major train; users could deploy this new release to get a subset of new features and changes with higher risk than would otherwise go into a minor release of the previous stable release train. If internally we wanted to add a new feature we could target this release while testing internally, and hopefully given the smaller delta between this “interim/experimental” to make it easier to re-base patches into the next major release train. This would help us avoid what today has unfortunately become a unavoidable large lag in getting new C* builds into production as while we attempt to sync our internal releases with a internally/or community QA’ed/accepted build/release of the current “stable” build/branch (currently this is 2.0).
>> 
>> To accomplish this, the commit workflow would unfortunately need change where an additional process is added to determine “eligibility” or “appropriateness” of a given commit to additionally also be committed to the “experimental” build branch (maybe it’s as simple as leaving it up to the reviewer + author to determine the risk factor and difficulty in merging the change back into the “experimental” build?). If it is agreed the commit/change/patch is a good candidate for the “experimental” branch, in addition to committing the patch to the current major release branch, the commit would also be merged into the new “experimental” release. If commits make it into the “experimental” branch frequently, I would expect/hope merging patches into the “experimental” build would be relatively easy as the “experimental” branch should also have most of the changes from the major release branch sans those considered highly risky or breaking. Additionally, if internally we want to work on a new feature and test internally before submitting a patch, we could target our code against the “experimental” branch, allowing us to test our changes in production without forking C* internally, writing our code against more recent “modern” changes, and then hopefully getting that work back to the community.
>> 
>> 
>> Hope this was clear enough and accurately summarizes the conversation a few of us had! Looking forward to everyone’s feedback and comments.
>> 
>> best,
>> kjellman