release policy

[Jonathan Ellis <jb...@gmail.com>]

I think we have enough people using Cassandra in production now that
it would be useful to be explicit about the kinds of changes we will
make between major and minor releases.  Here is one possibility:

Minor releases (e.g. 0.4.0 -> 0.4.1): minor releases will contain bug
fixes only -- no new functionality, full disk and network-level
compatibility.  If bugs cannot be fixed without adding new
functionality or breaking compatibility, then it should probably not
be fixed in the stable branch unless it is very severe.  This is
similar to postgresql's minor release policy, and I really like it for
two reasons:
  (1) it makes a clear line of what people can expect in minor releases
  (2) making changes breaks stuff.  That's just how software is, so
the less we change in a stable branch the more likely it is that we
won't introduce regressions.  With our community's most enthusiastic
testers also the most interested in trunk rather than older releases,
it behooves us to be careful.

Major releases (e.g. 0.5 -> 0.6): the compatibility story is going to
be more nuanced:
  (1) sstable compatibility: may change.  If it does, we will notify
you in NEWS.txt and provide some method of upgrading without
dump-and-reload (and hopefully without downtime while a conversion
tool grinds away from old to new format, but I don't think we can
promise that with 100% certainty at this point).
  (2) commitlog compatibility: may change.  If it does, we will notify
you in NEWS.txt that you need to "nodeprobe flush" before upgrading,
as was the case for 0.4 -> 0.5.
  (3) network compatibility (from one cluster node to another): may
change.  If it does, we will notify you in NEWS.txt that you need to
upgrade the whole cluster at once, as was the case for 0.4 -> 0.5.
  (4) thrift api: We will make our best efforts to keep deprecated
methods will be available for one major release before being removed.
(e.g., get_key_range is deprecated in 0.5; it will be removed in the
next).  I do not anticipate having to make the kind of changes we made
from 0.3 to 0.4 (where we redid basically all the thrift structures,
and there was really no sane way to provide backwards compatibility),
but if we do we will notify you in NEWS.txt.
  (5) StorageProxy "fat client" api: This is still considered "mostly
an internal api," to be used if you are brave and need the extra
performance badly enough. :)  We probably won't render it
unrecognizably different since it's a fairly one-to-one mapping to the
thrift api, which has the above compatibility policy, but no promises.

Thoughts?  (Separate threads to follow re "the next major release
after 0.5 specifically," and "1.0.")

-Jonathan

Re: release policy

[Ryan King <ry...@twitter.com>]

On Mon, Jan 11, 2010 at 10:15 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Mon, Jan 11, 2010 at 12:03 PM, Ryan King <ry...@twitter.com> wrote:
>> Both of the above are fine. I think we could even tolerate having to
>> run an upgrade tool with a node, as long as we can do it one at a time
>> and as long as...
>>
>>>  (3) network compatibility (from one cluster node to another): may
>>> change.  If it does, we will notify you in NEWS.txt that you need to
>>> upgrade the whole cluster at once, as was the case for 0.4 -> 0.5.
>>
>> ...the network compat doesn't change at the same time. If both the
>> disk format and network protocol change in the same release, we can't
>> easy do a rolling restart/upgrade.
>>
>> In general, doing full cluster upgrades at once is going to be
>> prohibitively difficult for us. In addition to the disruption it would
>> cause for clients, we wouldn't want to throw away all of our
>> in-process caches at once.
>
> I'd be comfortable committing to not breaking net compatibility in the
> same release as one that needs an upgrade tool to run on the data.  I
> don't think we can freeze the network protocol entirely yet.

Good.

>> If we want the flexibility of changing the internal network protocol,
>> we should move towards an rpc framework that can tolerate upgrades.
>
> Thrift is supposed to tolerate upgrades... *cough* :)

*cough* Avro *cough*.

-ryan

Re: release policy

[Jonathan Ellis <jb...@gmail.com>]

On Mon, Jan 11, 2010 at 12:03 PM, Ryan King <ry...@twitter.com> wrote:
> Both of the above are fine. I think we could even tolerate having to
> run an upgrade tool with a node, as long as we can do it one at a time
> and as long as...
>
>>  (3) network compatibility (from one cluster node to another): may
>> change.  If it does, we will notify you in NEWS.txt that you need to
>> upgrade the whole cluster at once, as was the case for 0.4 -> 0.5.
>
> ...the network compat doesn't change at the same time. If both the
> disk format and network protocol change in the same release, we can't
> easy do a rolling restart/upgrade.
>
> In general, doing full cluster upgrades at once is going to be
> prohibitively difficult for us. In addition to the disruption it would
> cause for clients, we wouldn't want to throw away all of our
> in-process caches at once.

I'd be comfortable committing to not breaking net compatibility in the
same release as one that needs an upgrade tool to run on the data.  I
don't think we can freeze the network protocol entirely yet.

> If we want the flexibility of changing the internal network protocol,
> we should move towards an rpc framework that can tolerate upgrades.

Thrift is supposed to tolerate upgrades... *cough* :)

-Jonathan

Re: release policy

[Ryan King <ry...@twitter.com>]

On Fri, Jan 8, 2010 at 11:47 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> I think we have enough people using Cassandra in production now that
> it would be useful to be explicit about the kinds of changes we will
> make between major and minor releases.  Here is one possibility:
>
> Minor releases (e.g. 0.4.0 -> 0.4.1): minor releases will contain bug
> fixes only -- no new functionality, full disk and network-level
> compatibility.  If bugs cannot be fixed without adding new
> functionality or breaking compatibility, then it should probably not
> be fixed in the stable branch unless it is very severe.  This is
> similar to postgresql's minor release policy, and I really like it for
> two reasons:
>  (1) it makes a clear line of what people can expect in minor releases
>  (2) making changes breaks stuff.  That's just how software is, so
> the less we change in a stable branch the more likely it is that we
> won't introduce regressions.  With our community's most enthusiastic
> testers also the most interested in trunk rather than older releases,
> it behooves us to be careful.
>
> Major releases (e.g. 0.5 -> 0.6): the compatibility story is going to
> be more nuanced:
>  (1) sstable compatibility: may change.  If it does, we will notify
> you in NEWS.txt and provide some method of upgrading without
> dump-and-reload (and hopefully without downtime while a conversion
> tool grinds away from old to new format, but I don't think we can
> promise that with 100% certainty at this point).
>  (2) commitlog compatibility: may change.  If it does, we will notify
> you in NEWS.txt that you need to "nodeprobe flush" before upgrading,
> as was the case for 0.4 -> 0.5.

Both of the above are fine. I think we could even tolerate having to
run an upgrade tool with a node, as long as we can do it one at a time
and as long as...

>  (3) network compatibility (from one cluster node to another): may
> change.  If it does, we will notify you in NEWS.txt that you need to
> upgrade the whole cluster at once, as was the case for 0.4 -> 0.5.

...the network compat doesn't change at the same time. If both the
disk format and network protocol change in the same release, we can't
easy do a rolling restart/upgrade.

In general, doing full cluster upgrades at once is going to be
prohibitively difficult for us. In addition to the disruption it would
cause for clients, we wouldn't want to throw away all of our
in-process caches at once.

If we want the flexibility of changing the internal network protocol,
we should move towards an rpc framework that can tolerate upgrades.

>  (4) thrift api: We will make our best efforts to keep deprecated
> methods will be available for one major release before being removed.
> (e.g., get_key_range is deprecated in 0.5; it will be removed in the
> next).  I do not anticipate having to make the kind of changes we made
> from 0.3 to 0.4 (where we redid basically all the thrift structures,
> and there was really no sane way to provide backwards compatibility),
> but if we do we will notify you in NEWS.txt.
>  (5) StorageProxy "fat client" api: This is still considered "mostly
> an internal api," to be used if you are brave and need the extra
> performance badly enough. :)  We probably won't render it
> unrecognizably different since it's a fairly one-to-one mapping to the
> thrift api, which has the above compatibility policy, but no promises.
>
> Thoughts?  (Separate threads to follow re "the next major release
> after 0.5 specifically," and "1.0.")
>
> -Jonathan
>

Re: release policy

[Chris Goffinet <cg...@chrisgoffinet.com>]

We maintain our own branch, but we have been good about rebasing changes
from 05 branch for last 2 months. We don't have major changes in our branch
(besides debian pkg and few custom Comparators and EndpointSnitches that
apply to our infrastructure) so the process has been fairly streamlined.
Internally we try to release as quickly as possible. We have experimental,
testing and prod clusters. We haven't been touching prod for awhile as we
gear up for a big Digg release with > 0.5.  We are now working on getting
trunk built in experimental and begin testing this weekend. We typically
apply patches before they even release 05/trunk and test in
experimental/testing direct from JIRA.

Once we go live with our big release, we've discussed using this exact
process to go from experimental, to testing to production. Obviously major
changes are going to require much more planning. It helps that we run in
multiple datacenters for some of these upgrades. Network changes that
require an entire cluster restart are going to be painful. We are hoping to
get some of the new SSTable changes in before our big release.

I agree with Jonathan's outline above for versioning.


On Fri, Jan 8, 2010 at 12:21 PM, Eric Evans <ee...@rackspace.com> wrote:

> On Fri, 2010-01-08 at 13:47 -0600, Jonathan Ellis wrote:
>
> [ ... ]
>
> > Thoughts?  (Separate threads to follow re "the next major release
> > after 0.5 specifically," and "1.0.")
>
> I'd be curious to hear from the Digg guys. My understanding is that
> they've been doing a considerable amount of custom release work,
> backporting patches into snapshots and the like. Is that still true?
> What would it take to remain more in-sync cassandra releases?
>
> --
> Eric Evans
> eevans@rackspace.com
>
>

Re: release policy

[Eric Evans <ee...@rackspace.com>]

On Fri, 2010-01-08 at 13:47 -0600, Jonathan Ellis wrote:

[ ... ]

> Thoughts?  (Separate threads to follow re "the next major release
> after 0.5 specifically," and "1.0.") 

I'd be curious to hear from the Digg guys. My understanding is that
they've been doing a considerable amount of custom release work,
backporting patches into snapshots and the like. Is that still true?
What would it take to remain more in-sync cassandra releases?

-- 
Eric Evans
eevans@rackspace.com