NEW: API Stability annotations!

[Gwen Shapira <gw...@confluent.io>]

Hi Team Kafka,

Ewen just added stability annotations to Apache Kafka (KAFKA-2429).

In the same PR, we marked the new Consumer API as "unstable" since we are
still actively iterating on them. The goal is to mark them as "evolving"
before the next release and after one release to validate them, we will
mark them as "stable".

When adding new public APIs, we encourage you to think of their stage of
development and annotate correctly.

The usage is:

- unstable - can change at any time
- evolving - can break compatibility at minor releases (i.e. 0.9 may be
incompatible with 0.8)
- stable - will only break compatibility at major releases (1.0, 2.0, etc)

Enjoy!

Gwen

Re: NEW: API Stability annotations!

[Edward Ribeiro <ed...@gmail.com>]

Very cool! Congrats! :)

My only comment is that, as Gwen pointed out in the first email, it is a
set of  API compatibility annotations, but the name is InterfaceStability.

Wouldn't ApiStability be a better name? InterfaceStability looks a bit
restrictive, imho.

https://github.com/apache/kafka/pull/133/files

Cheers,
Edward
Good annotations. I can see a few future usages :)

Jiangjie (Becket) Qin

On Wed, Aug 12, 2015 at 3:05 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Team Kafka,
>
> Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
>
> In the same PR, we marked the new Consumer API as "unstable" since we are
> still actively iterating on them. The goal is to mark them as "evolving"
> before the next release and after one release to validate them, we will
> mark them as "stable".
>
> When adding new public APIs, we encourage you to think of their stage of
> development and annotate correctly.
>
> The usage is:
>
> - unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>
> Enjoy!
>
> Gwen
>

Re: NEW: API Stability annotations!

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

Good annotations. I can see a few future usages :)

Jiangjie (Becket) Qin

On Wed, Aug 12, 2015 at 3:05 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Team Kafka,
>
> Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
>
> In the same PR, we marked the new Consumer API as "unstable" since we are
> still actively iterating on them. The goal is to mark them as "evolving"
> before the next release and after one release to validate them, we will
> mark them as "stable".
>
> When adding new public APIs, we encourage you to think of their stage of
> development and annotate correctly.
>
> The usage is:
>
> - unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>
> Enjoy!
>
> Gwen
>

Re: NEW: API Stability annotations!

[Gwen Shapira <gw...@confluent.io>]

Got it!
I was under the impression that deprecated means "there is something better
now and we'll remove this in few releases". But yeah, lets avoid marking it
for now.

On Thu, Aug 13, 2015 at 4:58 PM, Neha Narkhede <ne...@confluent.io> wrote:

> Thanks for starting the discussion around API annotations, Gwen! I think it
> is better to add the deprecated annotation after the new consumer API is
> successfully deployed at a couple places.
>
> On Thu, Aug 13, 2015 at 3:42 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >
> > > IMO, both old producer and old consumer APIs should be marked as
> > deprecated
> > > for 0.8.3 (since the new code will be in and we want to encourage the
> > > switch).
> > >
> >
> > I can see the appeal of this, but it's also worth considering the
> downsides
> > for users too:
> >
> > * It will introduce a number of deprecation warnings to everyone that
> > upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
> > worth keeping in mind that in many projects, warnings cause a build
> > failure)
> > * The new Consumer is still marked as `Unstable` so it seems a bit odd to
> > deprecate the old one
> >
> > I think a more conservative option would be to update the documentation
> to
> > encourage users to move to the new consumer without adding deprecation
> > annotations to the old consumer APIs. Some features that are only
> available
> > in the new consumer (e.g. SSL support) provide further incentive to move.
> > As Ewen suggested, the old consumer would then be deprecated in the
> > following release. And removed in the one after that. The main downside
> > would be having to maintain the old consumer for a little while longer.
> >
> > Something to think about. :)
> >
> > Best,
> > Ismael
> >
>
>
>
> --
> Thanks,
> Neha
>

Re: NEW: API Stability annotations!

[Neha Narkhede <ne...@confluent.io>]

Thanks for starting the discussion around API annotations, Gwen! I think it
is better to add the deprecated annotation after the new consumer API is
successfully deployed at a couple places.

On Thu, Aug 13, 2015 at 3:42 PM, Ismael Juma <is...@juma.me.uk> wrote:

> On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > IMO, both old producer and old consumer APIs should be marked as
> deprecated
> > for 0.8.3 (since the new code will be in and we want to encourage the
> > switch).
> >
>
> I can see the appeal of this, but it's also worth considering the downsides
> for users too:
>
> * It will introduce a number of deprecation warnings to everyone that
> upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
> worth keeping in mind that in many projects, warnings cause a build
> failure)
> * The new Consumer is still marked as `Unstable` so it seems a bit odd to
> deprecate the old one
>
> I think a more conservative option would be to update the documentation to
> encourage users to move to the new consumer without adding deprecation
> annotations to the old consumer APIs. Some features that are only available
> in the new consumer (e.g. SSL support) provide further incentive to move.
> As Ewen suggested, the old consumer would then be deprecated in the
> following release. And removed in the one after that. The main downside
> would be having to maintain the old consumer for a little while longer.
>
> Something to think about. :)
>
> Best,
> Ismael
>



-- 
Thanks,
Neha

Re: NEW: API Stability annotations!

[Ismael Juma <is...@juma.me.uk>]

On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira <gw...@confluent.io> wrote:

> IMO, both old producer and old consumer APIs should be marked as deprecated
> for 0.8.3 (since the new code will be in and we want to encourage the
> switch).
>

I can see the appeal of this, but it's also worth considering the downsides
for users too:

* It will introduce a number of deprecation warnings to everyone that
upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
worth keeping in mind that in many projects, warnings cause a build failure)
* The new Consumer is still marked as `Unstable` so it seems a bit odd to
deprecate the old one

I think a more conservative option would be to update the documentation to
encourage users to move to the new consumer without adding deprecation
annotations to the old consumer APIs. Some features that are only available
in the new consumer (e.g. SSL support) provide further incentive to move.
As Ewen suggested, the old consumer would then be deprecated in the
following release. And removed in the one after that. The main downside
would be having to maintain the old consumer for a little while longer.

Something to think about. :)

Best,
Ismael

Re: NEW: API Stability annotations!

[Gwen Shapira <gw...@confluent.io>]

IMO, both old producer and old consumer APIs should be marked as deprecated
for 0.8.3 (since the new code will be in and we want to encourage the
switch).

I'd like to see 0.9 released with just the new java clients.


On Thu, Aug 13, 2015 at 6:47 AM, Ismael Juma <is...@juma.me.uk> wrote:

> On Thu, Aug 13, 2015 at 8:36 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > On deprecation, I think we should definitely use the standard annotation
> to
> > handle this.
>
>
> Thanks Ewen. I agree that we should use @Deprecated for language-level
> deprecations. My question was regarding the following:
>
> - unstable - can change at any time
> > - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> > incompatible with 0.8)
> > - stable - will only break compatibility at major releases (1.0, 2.0,
> etc)
> >
>
>
> Let's pick `evolving`, as an example. Does it mean that we can break
> compatibility at minor releases without a deprecation cycle? Or does it
> mean that we would need to deprecate it for one minor release and then
> remove in the next? The latter is nicer for users as they have one release
> where both the deprecated and recommended still work, but it adds more
> overhead to Kafka development. What is the current thinking around this?
>
> The annotation doesn't necessarily work for everything (e.g.,
> > deprecating older versions of the protocol doesn't work via Java
> > deprecation annotations), but we should definitely use it for deprecated
> > Java/Scala APIs. Perhaps we should start by marking all the old producer
> > APIs deprecated
>
>
> Seems sensible assuming that users have found it easy to migrate to the new
> producer in the latest release.
>
> and, after the next release, the old consumer APIs too? :)
>
>
> Similar to above, it would be good to do that once we have some reports
> from users who moved from one API to the next.
>
> Best,
> Ismael
>

Re: NEW: API Stability annotations!

[Ismael Juma <is...@juma.me.uk>]

On Thu, Aug 13, 2015 at 8:36 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:

> On deprecation, I think we should definitely use the standard annotation to
> handle this.


Thanks Ewen. I agree that we should use @Deprecated for language-level
deprecations. My question was regarding the following:

- unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>


Let's pick `evolving`, as an example. Does it mean that we can break
compatibility at minor releases without a deprecation cycle? Or does it
mean that we would need to deprecate it for one minor release and then
remove in the next? The latter is nicer for users as they have one release
where both the deprecated and recommended still work, but it adds more
overhead to Kafka development. What is the current thinking around this?

The annotation doesn't necessarily work for everything (e.g.,
> deprecating older versions of the protocol doesn't work via Java
> deprecation annotations), but we should definitely use it for deprecated
> Java/Scala APIs. Perhaps we should start by marking all the old producer
> APIs deprecated


Seems sensible assuming that users have found it easy to migrate to the new
producer in the latest release.

and, after the next release, the old consumer APIs too? :)


Similar to above, it would be good to do that once we have some reports
from users who moved from one API to the next.

Best,
Ismael

Re: NEW: API Stability annotations!

[Ewen Cheslack-Postava <ew...@confluent.io>]

We could potentially add more annotations to cover things like
public/private APIs, deprecation, etc. I lifted this initial implementation
from Hadoop (thanks for the pointer Gwen!). Hadoop also has a
public/private annotation that we could lift, and they use this to filter
javadocs. These are useful signals, but we also don't want to make adding
new code too heavyweight, so I'd prefer keeping these annotations minimal.

The immediate benefit of adding stable/unstable annotations was to make it
clear a) that people should stop trying to use our in-progress APIs and b)
so it's clear when we're committing in-progress code that may not be as
good quality as we would normally require but is acceptable for new,
in-development code that we plan to quickly iterate and improve upon before
release. In the latter case, it's a useful signal internally for Kafka
development that an API isn't yet fully baked even for other Kafka devs.

On deprecation, I think we should definitely use the standard annotation to
handle this. The annotation doesn't necessarily work for everything (e.g.,
deprecating older versions of the protocol doesn't work via Java
deprecation annotations), but we should definitely use it for deprecated
Java/Scala APIs. Perhaps we should start by marking all the old producer
APIs deprecated and, after the next release, the old consumer APIs too? :)

-Ewen

On Wed, Aug 12, 2015 at 3:42 PM, Ismael Juma <is...@juma.me.uk> wrote:

> Hi Gwen,
>
> Nice to see this. There is no deprecation cycle for breakages?
>
> Ismael
>
> On Wed, Aug 12, 2015 at 11:05 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Hi Team Kafka,
> >
> > Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
> >
> > In the same PR, we marked the new Consumer API as "unstable" since we are
> > still actively iterating on them. The goal is to mark them as "evolving"
> > before the next release and after one release to validate them, we will
> > mark them as "stable".
> >
> > When adding new public APIs, we encourage you to think of their stage of
> > development and annotate correctly.
> >
> > The usage is:
> >
> > - unstable - can change at any time
> > - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> > incompatible with 0.8)
> > - stable - will only break compatibility at major releases (1.0, 2.0,
> etc)
> >
> > Enjoy!
> >
> > Gwen
> >
>



-- 
Thanks,
Ewen

Re: NEW: API Stability annotations!

[Ismael Juma <is...@juma.me.uk>]

Hi Gwen,

Nice to see this. There is no deprecation cycle for breakages?

Ismael

On Wed, Aug 12, 2015 at 11:05 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Team Kafka,
>
> Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
>
> In the same PR, we marked the new Consumer API as "unstable" since we are
> still actively iterating on them. The goal is to mark them as "evolving"
> before the next release and after one release to validate them, we will
> mark them as "stable".
>
> When adding new public APIs, we encourage you to think of their stage of
> development and annotate correctly.
>
> The usage is:
>
> - unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>
> Enjoy!
>
> Gwen
>

Re: NEW: API Stability annotations!

[Gwen Shapira <gw...@confluent.io>]

Yeah, I agree that your description matches reality better :)

By "document" do you have any suggestions where? The interface itself has
documentation. Were you thinking Wiki?

On Wed, Aug 12, 2015 at 3:48 PM, Grant Henke <gh...@cloudera.com> wrote:

> This is a great idea!
>
> I do question the classifications a bit. @stable seams unreasonable as
> Kafka has not has a "major" release yet. I have typically thought of Kafka
> releases as 0.{major}.{minor}.{maintenance}. For instance 0.8.3 coming up
> has a significant amount of change. Though Kafka's release cycle could be
> changing and I may not understand the cycle/classification correctly.
>
> I am thinking something along the lines of:
> - unstable - can change at any time (i.e maintenance releases)
> - evolving - can break compatibility at minor releases (i.e. 0.8.3 may
> be incompatible
> with 0.8.2)
> - stable - will only break compatibility at major releases (0.8, 0.9, etc)
>
> Regardless the choice of the "contract", I do suggest we document this.
>
>
> On Wed, Aug 12, 2015 at 5:05 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Hi Team Kafka,
> >
> > Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
> >
> > In the same PR, we marked the new Consumer API as "unstable" since we are
> > still actively iterating on them. The goal is to mark them as "evolving"
> > before the next release and after one release to validate them, we will
> > mark them as "stable".
> >
> > When adding new public APIs, we encourage you to think of their stage of
> > development and annotate correctly.
> >
> > The usage is:
> >
> > - unstable - can change at any time
> > - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> > incompatible with 0.8)
> > - stable - will only break compatibility at major releases (1.0, 2.0,
> etc)
> >
> > Enjoy!
> >
> > Gwen
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: NEW: API Stability annotations!

[Grant Henke <gh...@cloudera.com>]

This is a great idea!

I do question the classifications a bit. @stable seams unreasonable as
Kafka has not has a "major" release yet. I have typically thought of Kafka
releases as 0.{major}.{minor}.{maintenance}. For instance 0.8.3 coming up
has a significant amount of change. Though Kafka's release cycle could be
changing and I may not understand the cycle/classification correctly.

I am thinking something along the lines of:
- unstable - can change at any time (i.e maintenance releases)
- evolving - can break compatibility at minor releases (i.e. 0.8.3 may
be incompatible
with 0.8.2)
- stable - will only break compatibility at major releases (0.8, 0.9, etc)

Regardless the choice of the "contract", I do suggest we document this.


On Wed, Aug 12, 2015 at 5:05 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Team Kafka,
>
> Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
>
> In the same PR, we marked the new Consumer API as "unstable" since we are
> still actively iterating on them. The goal is to mark them as "evolving"
> before the next release and after one release to validate them, we will
> mark them as "stable".
>
> When adding new public APIs, we encourage you to think of their stage of
> development and annotate correctly.
>
> The usage is:
>
> - unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>
> Enjoy!
>
> Gwen
>



-- 
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke