Tracking backward-incompatible changes for Beam

[Dan Halperin <dh...@google.com.INVALID>]

Hey everyone,

In the Beam codebase, we’ve improved, rewritten, or deleted many APIs.
While this has improved the model and gives us great freedom to experiment,
we are also causing churn on users authoring Beam libraries and pipelines.

To really kick off Beam as something users can depend on, we need to
stabilize the Beam API. Stabilizing means a commitment to not making
breaking changes -- except between major versions as per standard semantic
versioning.

To get there, I’ve started a process for tracking these changes by applying
the `backward-incompatible` label [1] to the corresponding JIRA issues.
Naturally, open `backward-incompatible` changes are “blocking issues” for
the first stable release. (Or we’ll have to put them off for the next major
version!)

So here are some requests for help:
* Please review and appropriately label the components I skipped:
runner-{apex, flink, gearpump, spark}, sdk-py.
* Please proactively file JIRA issues for breaking API changes you still
want to make, and label them.

Thanks everyone!
Dan


[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible

Re: Tracking backward-incompatible changes for Beam

[Robert Bradshaw <ro...@google.com.INVALID>]

If the API/semantics are sufficiently well tested, backwards
incompatibility should manifest as test failures. The corollary is
that one should look closely at any test changes that get proposed.

On Mon, Oct 24, 2016 at 1:52 PM, Davor Bonaci <da...@apache.org> wrote:
> I don't think we have it right now. We should, of course, but this is
> something that needs to be defined/discussed first.
>
> On Mon, Oct 24, 2016 at 1:20 PM, Neelesh Salian <ns...@cloudera.com>
> wrote:
>
>> +1 for the labels and also a need for tests.
>> Do we document any rules for backward-compatibility? Be good to have a
>> checklist-like list.
>>
>>
>>
>>
>> On Mon, Oct 24, 2016 at 1:02 PM, Davor Bonaci <da...@google.com.invalid>
>> wrote:
>>
>> > It would be awesome to have that! At least a good portion of
>> > backward-incompatible changes could be automatically caught.
>> >
>> > We should also think about defining backward-compatibility more
>> precisely.
>> > This would be good in its own right, but also necessary to configure the
>> > tool. Historically, we have applied the backward-compatibility rules on
>> > APIs that are intended for users, excluding experimental ones, but not
>> > necessarily on all publicly visible APIs. If we continue this practice,
>> it
>> > might be a challenge for the tool. In any case, I think there's a good
>> > discussion to be had around what backward-compatibility means exactly in
>> > Beam.
>> >
>> > On Sat, Oct 22, 2016 at 2:47 AM, Aljoscha Krettek <al...@apache.org>
>> > wrote:
>> >
>> > > Very good idea!
>> > >
>> > > Should we already start thinking about automatic tests for backwards
>> > > compatibility of the API?
>> > >
>> > > On Fri, 21 Oct 2016 at 10:56 Jean-Baptiste Onofré <jb...@nanthrax.net>
>> > wrote:
>> > >
>> > > > Hi Dan,
>> > > >
>> > > > +1, good idea.
>> > > >
>> > > > Regards
>> > > > JB
>> > > >
>> > > > On 10/21/2016 02:21 AM, Dan Halperin wrote:
>> > > > > Hey everyone,
>> > > > >
>> > > > > In the Beam codebase, we’ve improved, rewritten, or deleted many
>> > APIs.
>> > > > > While this has improved the model and gives us great freedom to
>> > > > experiment,
>> > > > > we are also causing churn on users authoring Beam libraries and
>> > > > pipelines.
>> > > > >
>> > > > > To really kick off Beam as something users can depend on, we need
>> to
>> > > > > stabilize the Beam API. Stabilizing means a commitment to not
>> making
>> > > > > breaking changes -- except between major versions as per standard
>> > > > semantic
>> > > > > versioning.
>> > > > >
>> > > > > To get there, I’ve started a process for tracking these changes by
>> > > > applying
>> > > > > the `backward-incompatible` label [1] to the corresponding JIRA
>> > issues.
>> > > > > Naturally, open `backward-incompatible` changes are “blocking
>> issues”
>> > > for
>> > > > > the first stable release. (Or we’ll have to put them off for the
>> next
>> > > > major
>> > > > > version!)
>> > > > >
>> > > > > So here are some requests for help:
>> > > > > * Please review and appropriately label the components I skipped:
>> > > > > runner-{apex, flink, gearpump, spark}, sdk-py.
>> > > > > * Please proactively file JIRA issues for breaking API changes you
>> > > still
>> > > > > want to make, and label them.
>> > > > >
>> > > > > Thanks everyone!
>> > > > > Dan
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > https://issues.apache.org/jira/issues/?jql=project%20%
>> > > 3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
>> > > > >
>> > > >
>> > > > --
>> > > > Jean-Baptiste Onofré
>> > > > jbonofre@apache.org
>> > > > http://blog.nanthrax.net
>> > > > Talend - http://www.talend.com
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Neelesh Srinivas Salian
>> Customer Operations Engineer
>>

Re: Tracking backward-incompatible changes for Beam

[Davor Bonaci <da...@apache.org>]

I don't think we have it right now. We should, of course, but this is
something that needs to be defined/discussed first.

On Mon, Oct 24, 2016 at 1:20 PM, Neelesh Salian <ns...@cloudera.com>
wrote:

> +1 for the labels and also a need for tests.
> Do we document any rules for backward-compatibility? Be good to have a
> checklist-like list.
>
>
>
>
> On Mon, Oct 24, 2016 at 1:02 PM, Davor Bonaci <da...@google.com.invalid>
> wrote:
>
> > It would be awesome to have that! At least a good portion of
> > backward-incompatible changes could be automatically caught.
> >
> > We should also think about defining backward-compatibility more
> precisely.
> > This would be good in its own right, but also necessary to configure the
> > tool. Historically, we have applied the backward-compatibility rules on
> > APIs that are intended for users, excluding experimental ones, but not
> > necessarily on all publicly visible APIs. If we continue this practice,
> it
> > might be a challenge for the tool. In any case, I think there's a good
> > discussion to be had around what backward-compatibility means exactly in
> > Beam.
> >
> > On Sat, Oct 22, 2016 at 2:47 AM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > Very good idea!
> > >
> > > Should we already start thinking about automatic tests for backwards
> > > compatibility of the API?
> > >
> > > On Fri, 21 Oct 2016 at 10:56 Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > > Hi Dan,
> > > >
> > > > +1, good idea.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On 10/21/2016 02:21 AM, Dan Halperin wrote:
> > > > > Hey everyone,
> > > > >
> > > > > In the Beam codebase, we’ve improved, rewritten, or deleted many
> > APIs.
> > > > > While this has improved the model and gives us great freedom to
> > > > experiment,
> > > > > we are also causing churn on users authoring Beam libraries and
> > > > pipelines.
> > > > >
> > > > > To really kick off Beam as something users can depend on, we need
> to
> > > > > stabilize the Beam API. Stabilizing means a commitment to not
> making
> > > > > breaking changes -- except between major versions as per standard
> > > > semantic
> > > > > versioning.
> > > > >
> > > > > To get there, I’ve started a process for tracking these changes by
> > > > applying
> > > > > the `backward-incompatible` label [1] to the corresponding JIRA
> > issues.
> > > > > Naturally, open `backward-incompatible` changes are “blocking
> issues”
> > > for
> > > > > the first stable release. (Or we’ll have to put them off for the
> next
> > > > major
> > > > > version!)
> > > > >
> > > > > So here are some requests for help:
> > > > > * Please review and appropriately label the components I skipped:
> > > > > runner-{apex, flink, gearpump, spark}, sdk-py.
> > > > > * Please proactively file JIRA issues for breaking API changes you
> > > still
> > > > > want to make, and label them.
> > > > >
> > > > > Thanks everyone!
> > > > > Dan
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > https://issues.apache.org/jira/issues/?jql=project%20%
> > > 3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
> > > > >
> > > >
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbonofre@apache.org
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> >
>
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>

Re: Tracking backward-incompatible changes for Beam

[Neelesh Salian <ns...@cloudera.com>]

+1 for the labels and also a need for tests.
Do we document any rules for backward-compatibility? Be good to have a
checklist-like list.




On Mon, Oct 24, 2016 at 1:02 PM, Davor Bonaci <da...@google.com.invalid>
wrote:

> It would be awesome to have that! At least a good portion of
> backward-incompatible changes could be automatically caught.
>
> We should also think about defining backward-compatibility more precisely.
> This would be good in its own right, but also necessary to configure the
> tool. Historically, we have applied the backward-compatibility rules on
> APIs that are intended for users, excluding experimental ones, but not
> necessarily on all publicly visible APIs. If we continue this practice, it
> might be a challenge for the tool. In any case, I think there's a good
> discussion to be had around what backward-compatibility means exactly in
> Beam.
>
> On Sat, Oct 22, 2016 at 2:47 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > Very good idea!
> >
> > Should we already start thinking about automatic tests for backwards
> > compatibility of the API?
> >
> > On Fri, 21 Oct 2016 at 10:56 Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > > Hi Dan,
> > >
> > > +1, good idea.
> > >
> > > Regards
> > > JB
> > >
> > > On 10/21/2016 02:21 AM, Dan Halperin wrote:
> > > > Hey everyone,
> > > >
> > > > In the Beam codebase, we’ve improved, rewritten, or deleted many
> APIs.
> > > > While this has improved the model and gives us great freedom to
> > > experiment,
> > > > we are also causing churn on users authoring Beam libraries and
> > > pipelines.
> > > >
> > > > To really kick off Beam as something users can depend on, we need to
> > > > stabilize the Beam API. Stabilizing means a commitment to not making
> > > > breaking changes -- except between major versions as per standard
> > > semantic
> > > > versioning.
> > > >
> > > > To get there, I’ve started a process for tracking these changes by
> > > applying
> > > > the `backward-incompatible` label [1] to the corresponding JIRA
> issues.
> > > > Naturally, open `backward-incompatible` changes are “blocking issues”
> > for
> > > > the first stable release. (Or we’ll have to put them off for the next
> > > major
> > > > version!)
> > > >
> > > > So here are some requests for help:
> > > > * Please review and appropriately label the components I skipped:
> > > > runner-{apex, flink, gearpump, spark}, sdk-py.
> > > > * Please proactively file JIRA issues for breaking API changes you
> > still
> > > > want to make, and label them.
> > > >
> > > > Thanks everyone!
> > > > Dan
> > > >
> > > >
> > > > [1]
> > > >
> > > https://issues.apache.org/jira/issues/?jql=project%20%
> > 3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>



-- 
Neelesh Srinivas Salian
Customer Operations Engineer

Re: Tracking backward-incompatible changes for Beam

[Davor Bonaci <da...@google.com.INVALID>]

It would be awesome to have that! At least a good portion of
backward-incompatible changes could be automatically caught.

We should also think about defining backward-compatibility more precisely.
This would be good in its own right, but also necessary to configure the
tool. Historically, we have applied the backward-compatibility rules on
APIs that are intended for users, excluding experimental ones, but not
necessarily on all publicly visible APIs. If we continue this practice, it
might be a challenge for the tool. In any case, I think there's a good
discussion to be had around what backward-compatibility means exactly in
Beam.

On Sat, Oct 22, 2016 at 2:47 AM, Aljoscha Krettek <al...@apache.org>
wrote:

> Very good idea!
>
> Should we already start thinking about automatic tests for backwards
> compatibility of the API?
>
> On Fri, 21 Oct 2016 at 10:56 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Hi Dan,
> >
> > +1, good idea.
> >
> > Regards
> > JB
> >
> > On 10/21/2016 02:21 AM, Dan Halperin wrote:
> > > Hey everyone,
> > >
> > > In the Beam codebase, we’ve improved, rewritten, or deleted many APIs.
> > > While this has improved the model and gives us great freedom to
> > experiment,
> > > we are also causing churn on users authoring Beam libraries and
> > pipelines.
> > >
> > > To really kick off Beam as something users can depend on, we need to
> > > stabilize the Beam API. Stabilizing means a commitment to not making
> > > breaking changes -- except between major versions as per standard
> > semantic
> > > versioning.
> > >
> > > To get there, I’ve started a process for tracking these changes by
> > applying
> > > the `backward-incompatible` label [1] to the corresponding JIRA issues.
> > > Naturally, open `backward-incompatible` changes are “blocking issues”
> for
> > > the first stable release. (Or we’ll have to put them off for the next
> > major
> > > version!)
> > >
> > > So here are some requests for help:
> > > * Please review and appropriately label the components I skipped:
> > > runner-{apex, flink, gearpump, spark}, sdk-py.
> > > * Please proactively file JIRA issues for breaking API changes you
> still
> > > want to make, and label them.
> > >
> > > Thanks everyone!
> > > Dan
> > >
> > >
> > > [1]
> > >
> > https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Re: Tracking backward-incompatible changes for Beam

[Aljoscha Krettek <al...@apache.org>]

Very good idea!

Should we already start thinking about automatic tests for backwards
compatibility of the API?

On Fri, 21 Oct 2016 at 10:56 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi Dan,
>
> +1, good idea.
>
> Regards
> JB
>
> On 10/21/2016 02:21 AM, Dan Halperin wrote:
> > Hey everyone,
> >
> > In the Beam codebase, we’ve improved, rewritten, or deleted many APIs.
> > While this has improved the model and gives us great freedom to
> experiment,
> > we are also causing churn on users authoring Beam libraries and
> pipelines.
> >
> > To really kick off Beam as something users can depend on, we need to
> > stabilize the Beam API. Stabilizing means a commitment to not making
> > breaking changes -- except between major versions as per standard
> semantic
> > versioning.
> >
> > To get there, I’ve started a process for tracking these changes by
> applying
> > the `backward-incompatible` label [1] to the corresponding JIRA issues.
> > Naturally, open `backward-incompatible` changes are “blocking issues” for
> > the first stable release. (Or we’ll have to put them off for the next
> major
> > version!)
> >
> > So here are some requests for help:
> > * Please review and appropriately label the components I skipped:
> > runner-{apex, flink, gearpump, spark}, sdk-py.
> > * Please proactively file JIRA issues for breaking API changes you still
> > want to make, and label them.
> >
> > Thanks everyone!
> > Dan
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Tracking backward-incompatible changes for Beam

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Dan,

+1, good idea.

Regards
JB

On 10/21/2016 02:21 AM, Dan Halperin wrote:
> Hey everyone,
>
> In the Beam codebase, we\u2019ve improved, rewritten, or deleted many APIs.
> While this has improved the model and gives us great freedom to experiment,
> we are also causing churn on users authoring Beam libraries and pipelines.
>
> To really kick off Beam as something users can depend on, we need to
> stabilize the Beam API. Stabilizing means a commitment to not making
> breaking changes -- except between major versions as per standard semantic
> versioning.
>
> To get there, I\u2019ve started a process for tracking these changes by applying
> the `backward-incompatible` label [1] to the corresponding JIRA issues.
> Naturally, open `backward-incompatible` changes are \u201cblocking issues\u201d for
> the first stable release. (Or we\u2019ll have to put them off for the next major
> version!)
>
> So here are some requests for help:
> * Please review and appropriately label the components I skipped:
> runner-{apex, flink, gearpump, spark}, sdk-py.
> * Please proactively file JIRA issues for breaking API changes you still
> want to make, and label them.
>
> Thanks everyone!
> Dan
>
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Tracking backward-incompatible changes for Beam

[Ismaël Mejía <ie...@gmail.com>]

Thanks Dan, for putting this one in place, this one was so much needed in
particular as a first step to consolidate this info for users coming from
the Dataflow SDK or devs who follow only the releases.


On Fri, Oct 21, 2016 at 2:21 AM, Dan Halperin <dh...@google.com> wrote:

> Hey everyone,
>
> In the Beam codebase, we’ve improved, rewritten, or deleted many APIs.
> While this has improved the model and gives us great freedom to experiment,
> we are also causing churn on users authoring Beam libraries and pipelines.
>
> To really kick off Beam as something users can depend on, we need to
> stabilize the Beam API. Stabilizing means a commitment to not making
> breaking changes -- except between major versions as per standard semantic
> versioning.
>
> To get there, I’ve started a process for tracking these changes by
> applying the `backward-incompatible` label [1] to the corresponding JIRA
> issues. Naturally, open `backward-incompatible` changes are “blocking
> issues” for the first stable release. (Or we’ll have to put them off for
> the next major version!)
>
> So here are some requests for help:
> * Please review and appropriately label the components I skipped:
> runner-{apex, flink, gearpump, spark}, sdk-py.
> * Please proactively file JIRA issues for breaking API changes you still
> want to make, and label them.
>
> Thanks everyone!
> Dan
>
>
> [1] https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20BEAM%20AND%20labels%20%3D%20backward-incompatible
>
>