Next LTS?

[Austin Bennett <wh...@gmail.com>]

Hi All,

According to our policies page [1]: "There will be at least one new LTS
release in a 12 month period, and LTS releases are considered deprecated
after 12 months"

The last LTS was released 2018-10-02 [2].

Does that mean the next release (2.16) should be the next LTS?  It looks
like we are in danger of not living up to that promise.

Cheers,
Austin



[1] https://beam.apache.org/community/policies/

[2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Lukasz Cwik <lc...@google.com>]

I agree with what Lukasz Gajowy mentioned.

I find that Jenkins is fine when your developing at HEAD but as soon as you
cut a branch, the jenkins configuration starts to drift as it keeps getting
updated to HEAD with the seed job. I was always thinking that the Jenkins
configurations would be as thin as possible to invoke a single gradle
target that would be in the repo. This would allow for less drift since
most of the content would be coming from the repo itself.

On Fri, Sep 20, 2019 at 3:38 AM Łukasz Gajowy <lg...@apache.org> wrote:

> And, yes, the fact that Jenkins jobs are separately evolving but pretty
> tightly coupled to the repo contents is a serious problem that I wish we
> had fixed. So verification of each PR was manual.
>
> Could you tell a little bit more about what exactly were the problems back
> then? Was that due to incompatible docker images used in portability tests
> maybe? Any other issues?
>
> My thoughts about Jenkins revolved around decoupling from Jenkins plugins,
> Groovy DSL etc and replace as much as possible with more universal tools
> (bash, Gradle). The main drivers were to: (1) be able to run the same thing
> that runs on Jenkins using bash/Gradle (same scripts), (2) potentially be
> able to replace Jenkins more easily with some more modern/better CI/CD tool
> in the future (Github Actions/Gitlab or simply newer Jenkins with
> Jenkinsfiles). I don't understand yet what was the problem cited above (I
> didn't work on the LTS back then) so I'm not sure it would help with
> releasing LTS versions with backports.
>
> Łukasz
>
> pt., 20 wrz 2019 o 02:01 Ahmet Altay <al...@google.com> napisał(a):
>
>> I agree with retiring 2.7 as the LTS family. Based on my experience with
>> users 2.7 does not have a particularly high adoption and as pointed out has
>> known critical issues. Declaring another LTS pending demand sounds
>> reasonable but how are we going to gauge this demand?
>>
>> +Yifan Zou <yi...@google.com> +Alan Myrvold <am...@google.com> on
>> the tooling question as well. Unless we address the tooling problem it
>> seems difficult to feasibly maintain LTS versions over time.
>>
>> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>>
>>> To be clear, I was picking on - or reminding us of - the promise: I
>>> don't have a strong personal need/desire (at least currently) for LTS to
>>> exist.  Though, worth ensuring we live up to what we keep on the website.
>>> And, without an active LTS, probably something we should take off the site?
>>>
>>>
>>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> +Łukasz Gajowy <lu...@polidea.com> had at some point thought
>>>> of setting up jenkins jobs without coupling them to the state of the repo
>>>> during the last Seed Job. It may be that that improvement can help test
>>>> older LTS-type releases?
>>>>
>>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>>
>>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>>>> we learned some valuable lessons. It would have been good to push out
>>>>> something (even if it didn't have everything we wanted) but that is
>>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>>>> as LTS and no longer recommended).
>>>>>
>>>>> I agree that it does not seem there is strong demand for an LTS at
>>>>> this point. I would propose that we keep 2.16, etc. as potential
>>>>> candidates, but only declare one as LTS pending demand. The question
>>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>>>> could itself be a driver for an LTS).
>>>>>
>>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>>>>> interest.
>>>>> >
>>>>> > There are known problems so that I cannot recommend anyone to use
>>>>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>>>>> philosophical. I did not like the fact that we had a designated LTS family
>>>>> with no usable releases.
>>>>> >
>>>>> > But many backports were proposed to block 2.7.1 and took a very long
>>>>> time to get contirbutors to implement the backports. I ended up doing many
>>>>> of them just to move it along. This indicates a lack of interest to me. The
>>>>> problem is that we cannot really use a strict cut off date as a way to
>>>>> ensure people do the important things and skip the unimportant things,
>>>>> because we do know that the issues are critical.
>>>>> >
>>>>> > And, yes, the fact that Jenkins jobs are separately evolving but
>>>>> pretty tightly coupled to the repo contents is a serious problem that I
>>>>> wish we had fixed. So verification of each PR was manual.
>>>>> >
>>>>> > Altogether, I still think LTS is valuable to have as a promise to
>>>>> users that we will backport critical fixes. I would like to keep that
>>>>> promise and continue to try. Things that are rapidly changing (which
>>>>> something always will be) just won't have fixes backported, and that seems
>>>>> OK.
>>>>> >
>>>>> > Kenn
>>>>> >
>>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
>>>>> wrote:
>>>>> >>
>>>>> >> An LTS only makes sense if we end up patching the LTS, which so far
>>>>> we
>>>>> >> have never done. There has been work done in backporting fixes, see
>>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the
>>>>> effort was
>>>>> >> never completed. The main reason I believe were complications with
>>>>> >> running the evolved release scripts against old Beam versions.
>>>>> >>
>>>>> >> Now that the portability layer keeps maturing, it makes me
>>>>> optimistic
>>>>> >> that we might have a maintained LTS in the future.
>>>>> >>
>>>>> >> -Max
>>>>> >>
>>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS
>>>>> and for
>>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>>>> >> > interest on having a LTS.
>>>>> >> >
>>>>> >> > We still are heavily iterating in many areas (portability/schema)
>>>>> and
>>>>> >> > I am not sure users (and in particular users of open source
>>>>> runners)
>>>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>>>> >> > moment to reconsider if having a LTS does even make sense given
>>>>> (1)
>>>>> >> > that our end user facing APIs are 'mostly' stable (even if many
>>>>> still
>>>>> >> > called @Experimental). (2) that users get mostly improvements on
>>>>> >> > runners translation and newer APIs with a low cost just by
>>>>> updating
>>>>> >> > the version number, and (3) that in case of any regression in an
>>>>> >> > intermediary release we still can do a minor release even if we
>>>>> have
>>>>> >> > not yet done so, let's not forget that the only thing we need to
>>>>> do
>>>>> >> > this is enough interest to do the release from the maintainers.
>>>>> >> >
>>>>> >> >
>>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>>>> >> > <va...@google.com> wrote:
>>>>> >> >>
>>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust
>>>>> Python 3 support compared with prior releases, and also for reasons of
>>>>> pending Python 2 deprecation. This has been discussed before [1]. As Robert
>>>>> pointed out in that thread, LTS nomination in Beam is currently
>>>>> retroactive. If we keep the retroactive policy, the question is how long we
>>>>> should wait for a release to be considered "safe" for nomination.  Looks
>>>>> like in case of 2.7.0 we waited a month, see [2,3].
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >> Valentyn
>>>>> >> >>
>>>>> >> >> [1]
>>>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>>>> >> >> [3]
>>>>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>>>> >> >>
>>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>>>>> whatwouldaustindo@gmail.com> wrote:
>>>>> >> >>>
>>>>> >> >>> Hi All,
>>>>> >> >>>
>>>>> >> >>> According to our policies page [1]: "There will be at least one
>>>>> new LTS release in a 12 month period, and LTS releases are considered
>>>>> deprecated after 12 months"
>>>>> >> >>>
>>>>> >> >>> The last LTS was released 2018-10-02 [2].
>>>>> >> >>>
>>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?
>>>>> It looks like we are in danger of not living up to that promise.
>>>>> >> >>>
>>>>> >> >>> Cheers,
>>>>> >> >>> Austin
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>> [1] https://beam.apache.org/community/policies/
>>>>> >> >>>
>>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>>>>
>>>>

Re: Next LTS?

[Łukasz Gajowy <lg...@apache.org>]

And, yes, the fact that Jenkins jobs are separately evolving but pretty
tightly coupled to the repo contents is a serious problem that I wish we
had fixed. So verification of each PR was manual.

Could you tell a little bit more about what exactly were the problems back
then? Was that due to incompatible docker images used in portability tests
maybe? Any other issues?

My thoughts about Jenkins revolved around decoupling from Jenkins plugins,
Groovy DSL etc and replace as much as possible with more universal tools
(bash, Gradle). The main drivers were to: (1) be able to run the same thing
that runs on Jenkins using bash/Gradle (same scripts), (2) potentially be
able to replace Jenkins more easily with some more modern/better CI/CD tool
in the future (Github Actions/Gitlab or simply newer Jenkins with
Jenkinsfiles). I don't understand yet what was the problem cited above (I
didn't work on the LTS back then) so I'm not sure it would help with
releasing LTS versions with backports.

Łukasz

pt., 20 wrz 2019 o 02:01 Ahmet Altay <al...@google.com> napisał(a):

> I agree with retiring 2.7 as the LTS family. Based on my experience with
> users 2.7 does not have a particularly high adoption and as pointed out has
> known critical issues. Declaring another LTS pending demand sounds
> reasonable but how are we going to gauge this demand?
>
> +Yifan Zou <yi...@google.com> +Alan Myrvold <am...@google.com> on
> the tooling question as well. Unless we address the tooling problem it
> seems difficult to feasibly maintain LTS versions over time.
>
> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
>
>> To be clear, I was picking on - or reminding us of - the promise: I don't
>> have a strong personal need/desire (at least currently) for LTS to exist.
>> Though, worth ensuring we live up to what we keep on the website.  And,
>> without an active LTS, probably something we should take off the site?
>>
>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> +Łukasz Gajowy <lu...@polidea.com> had at some point thought of
>>> setting up jenkins jobs without coupling them to the state of the repo
>>> during the last Seed Job. It may be that that improvement can help test
>>> older LTS-type releases?
>>>
>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>>> we learned some valuable lessons. It would have been good to push out
>>>> something (even if it didn't have everything we wanted) but that is
>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>>> as LTS and no longer recommended).
>>>>
>>>> I agree that it does not seem there is strong demand for an LTS at
>>>> this point. I would propose that we keep 2.16, etc. as potential
>>>> candidates, but only declare one as LTS pending demand. The question
>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>>> could itself be a driver for an LTS).
>>>>
>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
>>>> wrote:
>>>> >
>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>>>> interest.
>>>> >
>>>> > There are known problems so that I cannot recommend anyone to use
>>>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>>>> philosophical. I did not like the fact that we had a designated LTS family
>>>> with no usable releases.
>>>> >
>>>> > But many backports were proposed to block 2.7.1 and took a very long
>>>> time to get contirbutors to implement the backports. I ended up doing many
>>>> of them just to move it along. This indicates a lack of interest to me. The
>>>> problem is that we cannot really use a strict cut off date as a way to
>>>> ensure people do the important things and skip the unimportant things,
>>>> because we do know that the issues are critical.
>>>> >
>>>> > And, yes, the fact that Jenkins jobs are separately evolving but
>>>> pretty tightly coupled to the repo contents is a serious problem that I
>>>> wish we had fixed. So verification of each PR was manual.
>>>> >
>>>> > Altogether, I still think LTS is valuable to have as a promise to
>>>> users that we will backport critical fixes. I would like to keep that
>>>> promise and continue to try. Things that are rapidly changing (which
>>>> something always will be) just won't have fixes backported, and that seems
>>>> OK.
>>>> >
>>>> > Kenn
>>>> >
>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> An LTS only makes sense if we end up patching the LTS, which so far
>>>> we
>>>> >> have never done. There has been work done in backporting fixes, see
>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort
>>>> was
>>>> >> never completed. The main reason I believe were complications with
>>>> >> running the evolved release scripts against old Beam versions.
>>>> >>
>>>> >> Now that the portability layer keeps maturing, it makes me optimistic
>>>> >> that we might have a maintained LTS in the future.
>>>> >>
>>>> >> -Max
>>>> >>
>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and
>>>> for
>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>>> >> > interest on having a LTS.
>>>> >> >
>>>> >> > We still are heavily iterating in many areas (portability/schema)
>>>> and
>>>> >> > I am not sure users (and in particular users of open source
>>>> runners)
>>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>>> >> > moment to reconsider if having a LTS does even make sense given (1)
>>>> >> > that our end user facing APIs are 'mostly' stable (even if many
>>>> still
>>>> >> > called @Experimental). (2) that users get mostly improvements on
>>>> >> > runners translation and newer APIs with a low cost just by updating
>>>> >> > the version number, and (3) that in case of any regression in an
>>>> >> > intermediary release we still can do a minor release even if we
>>>> have
>>>> >> > not yet done so, let's not forget that the only thing we need to do
>>>> >> > this is enough interest to do the release from the maintainers.
>>>> >> >
>>>> >> >
>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>>> >> > <va...@google.com> wrote:
>>>> >> >>
>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust
>>>> Python 3 support compared with prior releases, and also for reasons of
>>>> pending Python 2 deprecation. This has been discussed before [1]. As Robert
>>>> pointed out in that thread, LTS nomination in Beam is currently
>>>> retroactive. If we keep the retroactive policy, the question is how long we
>>>> should wait for a release to be considered "safe" for nomination.  Looks
>>>> like in case of 2.7.0 we waited a month, see [2,3].
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Valentyn
>>>> >> >>
>>>> >> >> [1]
>>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>>> >> >> [3]
>>>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>>> >> >>
>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>>>> whatwouldaustindo@gmail.com> wrote:
>>>> >> >>>
>>>> >> >>> Hi All,
>>>> >> >>>
>>>> >> >>> According to our policies page [1]: "There will be at least one
>>>> new LTS release in a 12 month period, and LTS releases are considered
>>>> deprecated after 12 months"
>>>> >> >>>
>>>> >> >>> The last LTS was released 2018-10-02 [2].
>>>> >> >>>
>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?
>>>> It looks like we are in danger of not living up to that promise.
>>>> >> >>>
>>>> >> >>> Cheers,
>>>> >> >>> Austin
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> [1] https://beam.apache.org/community/policies/
>>>> >> >>>
>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>>>
>>>

Re: Next LTS?

[Ahmet Altay <al...@google.com>]

On Wed, Mar 25, 2020 at 8:35 AM Austin Bennett <wh...@gmail.com>
wrote:

> I'll submit PR, and tag you, Ismael.
>
> Merge once there's a sufficient consensus.
>
>
> On Wed, Mar 25, 2020 at 3:57 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> +1 to remove any mention of LTS related information to not create
>> misinformation on users.
>> Would you be up to do that Austin? or someone else?
>>
>> On Wed, Mar 25, 2020 at 1:02 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >
>> > I would want to avoid maintain a Python 2 LTS, even if just for the
>> > fact that the infrastructure might not be there.
>> >
>> > On Tue, Mar 24, 2020 at 3:58 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> > >
>> > > Yes, we had a suggestion to pick a stable Python 2 release as an LTS.
>> The suggestion assumed that LTS will continue to exist. Now, if Python 2 is
>> the only reason to have an LTS, we can consider it as long as:
>> > > - we scope the LTS portion to Python SDK only.
>> > > - we have an ownership story for Python 2 LTS, for example volunteers
>> in dev or user community who will be willing to maintain that release.
>> > >
>> > > We can bring this up when we drop Python 2 support. We decided to
>> revisit that conversation in a couple of months IIRC.
>> > >
>> > > On Tue, Mar 24, 2020 at 3:44 PM Ahmet Altay <al...@google.com> wrote:
>> > >>
>> > >> Removing it makes sense. We did not have a good way of measuring the
>> demand for LTS releases.
>> > >>
>> > >> There was a suggestion to mark the last release with python 2
>> support to be an LTS release, was there a conclusion on that? ( +Valentyn
>> Tymofieiev )
>> > >>
>> > >> Ahmet
>> > >>
>> > >> On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> > >>>
>> > >>> There seems to have been lack of demand. I agree we should remove
>> > >>> these statements from our site until we find a reason to re-visit
>> > >>> doing LTS release.
>> > >>>
>> > >>> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
>> > >>> <wh...@gmail.com> wrote:
>> > >>> >
>> > >>> > What's our LTS policy these days?  It seems we should remove the
>> following from our site (and encourage GCP does the same, below), if we're
>> not going to maintain these.  I'll update policy page via PR, if get the go
>> ahead that it is our desire.  Seems we can't suggest policies in a policy
>> doc that we don't follow...?
>> > >>> >
>> > >>> > I am not trying to suggest demand for LTS.  If others haven't
>> spoken up, that also indicates lack of demand.  Point of my message is to
>> say, we should update our Policies doc, if those aren't what we are
>> practicing (and can re-add later if wanting to revive LTS).
>> > >>> >
>> > >>> > https://beam.apache.org/community/policies/
>> > >>> >
>> > >>> > Apache Beam aims to make 8 releases in a 12 month period. To
>> accommodate users with longer upgrade cycles, some of these releases will
>> be tagged as long term support (LTS) releases. LTS releases receive patches
>> to fix major issues for 12 months, starting from the release’s initial
>> release date. There will be at least one new LTS release in a 12 month
>> period, and LTS releases are considered deprecated after 12 months. The
>> community will mark a release as a LTS release based on various factors,
>> such as the number of LTS releases currently in flight and whether the
>> accumulated feature set since the last LTS provides significant upgrade
>> value. Non-LTS releases do not receive patches and are considered
>> deprecated immediately after the next following minor release. We encourage
>> you to update early and often; do not wait until the deprecation date of
>> the version you are using.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Seems a Google Specific Concern, but related to the community:
>> https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
>> > >>> >
>> > >>> > Apache Beam is an open source, community-led project. Google is
>> part of the community, but we do not own the project or control the release
>> process. We might open bugs or submit patches to the Apache Beam codebase
>> on behalf of Dataflow customers, but we cannot create hotfixes or official
>> releases of Apache Beam on demand.
>> > >>> >
>> > >>> > However, the Apache Beam community designates specific releases
>> as long term support (LTS) releases. LTS releases receive patches to fix
>> major issues for a designated period of time. See the Apache Beam policies
>> page for more details about release policies.
>>
>
Adding +Rose Nguyen <rt...@google.com> explicitly to make follow up
changes to the dataflow specific documentation mentioned here after
Austin's PR to change Beam docs.


> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com>
>> wrote:
>> > >>> >>
>> > >>> >> I agree with retiring 2.7 as the LTS family. Based on my
>> experience with users 2.7 does not have a particularly high adoption and as
>> pointed out has known critical issues. Declaring another LTS pending demand
>> sounds reasonable but how are we going to gauge this demand?
>> > >>> >>
>> > >>> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless
>> we address the tooling problem it seems difficult to feasibly maintain LTS
>> versions over time.
>> > >>> >>
>> > >>> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>> > >>> >>>
>> > >>> >>> To be clear, I was picking on - or reminding us of - the
>> promise: I don't have a strong personal need/desire (at least currently)
>> for LTS to exist.  Though, worth ensuring we live up to what we keep on the
>> website.  And, without an active LTS, probably something we should take off
>> the site?
>> > >>> >>>
>> > >>> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <
>> pabloem@google.com> wrote:
>> > >>> >>>>
>> > >>> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins
>> jobs without coupling them to the state of the repo during the last Seed
>> Job. It may be that that improvement can help test older LTS-type releases?
>> > >>> >>>>
>> > >>> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> > >>> >>>>>
>> > >>> >>>>> In many ways the 2.7 LTS was trying to flesh out the process.
>> I think
>> > >>> >>>>> we learned some valuable lessons. It would have been good to
>> push out
>> > >>> >>>>> something (even if it didn't have everything we wanted) but
>> that is
>> > >>> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be
>> retired
>> > >>> >>>>> as LTS and no longer recommended).
>> > >>> >>>>>
>> > >>> >>>>> I agree that it does not seem there is strong demand for an
>> LTS at
>> > >>> >>>>> this point. I would propose that we keep 2.16, etc. as
>> potential
>> > >>> >>>>> candidates, but only declare one as LTS pending demand. The
>> question
>> > >>> >>>>> of how to keep our tooling stable (or backwards/forwards
>> compatible)
>> > >>> >>>>> is a good one, especially as we move to drop Python 2.7 in
>> 2020 (which
>> > >>> >>>>> could itself be a driver for an LTS).
>> > >>> >>>>>
>> > >>> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <
>> kenn@apache.org> wrote:
>> > >>> >>>>> >
>> > >>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to
>> lack of interest.
>> > >>> >>>>> >
>> > >>> >>>>> > There are known problems so that I cannot recommend anyone
>> to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>> philosophical. I did not like the fact that we had a designated LTS family
>> with no usable releases.
>> > >>> >>>>> >
>> > >>> >>>>> > But many backports were proposed to block 2.7.1 and took a
>> very long time to get contirbutors to implement the backports. I ended up
>> doing many of them just to move it along. This indicates a lack of interest
>> to me. The problem is that we cannot really use a strict cut off date as a
>> way to ensure people do the important things and skip the unimportant
>> things, because we do know that the issues are critical.
>> > >>> >>>>> >
>> > >>> >>>>> > And, yes, the fact that Jenkins jobs are separately
>> evolving but pretty tightly coupled to the repo contents is a serious
>> problem that I wish we had fixed. So verification of each PR was manual.
>> > >>> >>>>> >
>> > >>> >>>>> > Altogether, I still think LTS is valuable to have as a
>> promise to users that we will backport critical fixes. I would like to keep
>> that promise and continue to try. Things that are rapidly changing (which
>> something always will be) just won't have fixes backported, and that seems
>> OK.
>> > >>> >>>>> >
>> > >>> >>>>> > Kenn
>> > >>> >>>>> >
>> > >>> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <
>> mxm@apache.org> wrote:
>> > >>> >>>>> >>
>> > >>> >>>>> >> An LTS only makes sense if we end up patching the LTS,
>> which so far we
>> > >>> >>>>> >> have never done. There has been work done in backporting
>> fixes, see
>> > >>> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but
>> the effort was
>> > >>> >>>>> >> never completed. The main reason I believe were
>> complications with
>> > >>> >>>>> >> running the evolved release scripts against old Beam
>> versions.
>> > >>> >>>>> >>
>> > >>> >>>>> >> Now that the portability layer keeps maturing, it makes me
>> optimistic
>> > >>> >>>>> >> that we might have a maintained LTS in the future.
>> > >>> >>>>> >>
>> > >>> >>>>> >> -Max
>> > >>> >>>>> >>
>> > >>> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>> > >>> >>>>> >> > The fact that end users never asked AFAIK in the ML for
>> an LTS and for
>> > >>> >>>>> >> > a subsequent minor release of the existing LTS shows IMO
>> the low
>> > >>> >>>>> >> > interest on having a LTS.
>> > >>> >>>>> >> >
>> > >>> >>>>> >> > We still are heavily iterating in many areas
>> (portability/schema) and
>> > >>> >>>>> >> > I am not sure users (and in particular users of open
>> source runners)
>> > >>> >>>>> >> > get a big benefit of relying on an old version. Maybe
>> this is the
>> > >>> >>>>> >> > moment to reconsider if having a LTS does even make
>> sense given (1)
>> > >>> >>>>> >> > that our end user facing APIs are 'mostly' stable (even
>> if many still
>> > >>> >>>>> >> > called @Experimental). (2) that users get mostly
>> improvements on
>> > >>> >>>>> >> > runners translation and newer APIs with a low cost just
>> by updating
>> > >>> >>>>> >> > the version number, and (3) that in case of any
>> regression in an
>> > >>> >>>>> >> > intermediary release we still can do a minor release
>> even if we have
>> > >>> >>>>> >> > not yet done so, let's not forget that the only thing we
>> need to do
>> > >>> >>>>> >> > this is enough interest to do the release from the
>> maintainers.
>> > >>> >>>>> >> >
>> > >>> >>>>> >> >
>> > >>> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>> > >>> >>>>> >> > <va...@google.com> wrote:
>> > >>> >>>>> >> >>
>> > >>> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has
>> robust Python 3 support compared with prior releases, and also for reasons
>> of pending Python 2 deprecation. This has been discussed before [1]. As
>> Robert pointed out in that thread, LTS nomination in Beam is currently
>> retroactive. If we keep the retroactive policy, the question is how long we
>> should wait for a release to be considered "safe" for nomination.  Looks
>> like in case of 2.7.0 we waited a month, see [2,3].
>> > >>> >>>>> >> >>
>> > >>> >>>>> >> >> Thanks,
>> > >>> >>>>> >> >> Valentyn
>> > >>> >>>>> >> >>
>> > >>> >>>>> >> >> [1]
>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> > >>> >>>>> >> >> [2]
>> https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>> > >>> >>>>> >> >> [3]
>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>> > >>> >>>>> >> >>
>> > >>> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> Hi All,
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> According to our policies page [1]: "There will be at
>> least one new LTS release in a 12 month period, and LTS releases are
>> considered deprecated after 12 months"
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> Does that mean the next release (2.16) should be the
>> next LTS?  It looks like we are in danger of not living up to that promise.
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> Cheers,
>> > >>> >>>>> >> >>> Austin
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
>> > >>> >>>>> >> >>>
>> > >>> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>
>

Re: Next LTS?

[Austin Bennett <wh...@gmail.com>]

I'll submit PR, and tag you, Ismael.

Merge once there's a sufficient consensus.


On Wed, Mar 25, 2020 at 3:57 AM Ismaël Mejía <ie...@gmail.com> wrote:

> +1 to remove any mention of LTS related information to not create
> misinformation on users.
> Would you be up to do that Austin? or someone else?
>
> On Wed, Mar 25, 2020 at 1:02 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >
> > I would want to avoid maintain a Python 2 LTS, even if just for the
> > fact that the infrastructure might not be there.
> >
> > On Tue, Mar 24, 2020 at 3:58 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
> > >
> > > Yes, we had a suggestion to pick a stable Python 2 release as an LTS.
> The suggestion assumed that LTS will continue to exist. Now, if Python 2 is
> the only reason to have an LTS, we can consider it as long as:
> > > - we scope the LTS portion to Python SDK only.
> > > - we have an ownership story for Python 2 LTS, for example volunteers
> in dev or user community who will be willing to maintain that release.
> > >
> > > We can bring this up when we drop Python 2 support. We decided to
> revisit that conversation in a couple of months IIRC.
> > >
> > > On Tue, Mar 24, 2020 at 3:44 PM Ahmet Altay <al...@google.com> wrote:
> > >>
> > >> Removing it makes sense. We did not have a good way of measuring the
> demand for LTS releases.
> > >>
> > >> There was a suggestion to mark the last release with python 2 support
> to be an LTS release, was there a conclusion on that? ( +Valentyn
> Tymofieiev )
> > >>
> > >> Ahmet
> > >>
> > >> On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com>
> wrote:
> > >>>
> > >>> There seems to have been lack of demand. I agree we should remove
> > >>> these statements from our site until we find a reason to re-visit
> > >>> doing LTS release.
> > >>>
> > >>> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
> > >>> <wh...@gmail.com> wrote:
> > >>> >
> > >>> > What's our LTS policy these days?  It seems we should remove the
> following from our site (and encourage GCP does the same, below), if we're
> not going to maintain these.  I'll update policy page via PR, if get the go
> ahead that it is our desire.  Seems we can't suggest policies in a policy
> doc that we don't follow...?
> > >>> >
> > >>> > I am not trying to suggest demand for LTS.  If others haven't
> spoken up, that also indicates lack of demand.  Point of my message is to
> say, we should update our Policies doc, if those aren't what we are
> practicing (and can re-add later if wanting to revive LTS).
> > >>> >
> > >>> > https://beam.apache.org/community/policies/
> > >>> >
> > >>> > Apache Beam aims to make 8 releases in a 12 month period. To
> accommodate users with longer upgrade cycles, some of these releases will
> be tagged as long term support (LTS) releases. LTS releases receive patches
> to fix major issues for 12 months, starting from the release’s initial
> release date. There will be at least one new LTS release in a 12 month
> period, and LTS releases are considered deprecated after 12 months. The
> community will mark a release as a LTS release based on various factors,
> such as the number of LTS releases currently in flight and whether the
> accumulated feature set since the last LTS provides significant upgrade
> value. Non-LTS releases do not receive patches and are considered
> deprecated immediately after the next following minor release. We encourage
> you to update early and often; do not wait until the deprecation date of
> the version you are using.
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > Seems a Google Specific Concern, but related to the community:
> https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
> > >>> >
> > >>> > Apache Beam is an open source, community-led project. Google is
> part of the community, but we do not own the project or control the release
> process. We might open bugs or submit patches to the Apache Beam codebase
> on behalf of Dataflow customers, but we cannot create hotfixes or official
> releases of Apache Beam on demand.
> > >>> >
> > >>> > However, the Apache Beam community designates specific releases as
> long term support (LTS) releases. LTS releases receive patches to fix major
> issues for a designated period of time. See the Apache Beam policies page
> for more details about release policies.
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com>
> wrote:
> > >>> >>
> > >>> >> I agree with retiring 2.7 as the LTS family. Based on my
> experience with users 2.7 does not have a particularly high adoption and as
> pointed out has known critical issues. Declaring another LTS pending demand
> sounds reasonable but how are we going to gauge this demand?
> > >>> >>
> > >>> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless
> we address the tooling problem it seems difficult to feasibly maintain LTS
> versions over time.
> > >>> >>
> > >>> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> > >>> >>>
> > >>> >>> To be clear, I was picking on - or reminding us of - the
> promise: I don't have a strong personal need/desire (at least currently)
> for LTS to exist.  Though, worth ensuring we live up to what we keep on the
> website.  And, without an active LTS, probably something we should take off
> the site?
> > >>> >>>
> > >>> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <
> pabloem@google.com> wrote:
> > >>> >>>>
> > >>> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins
> jobs without coupling them to the state of the repo during the last Seed
> Job. It may be that that improvement can help test older LTS-type releases?
> > >>> >>>>
> > >>> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> > >>> >>>>>
> > >>> >>>>> In many ways the 2.7 LTS was trying to flesh out the process.
> I think
> > >>> >>>>> we learned some valuable lessons. It would have been good to
> push out
> > >>> >>>>> something (even if it didn't have everything we wanted) but
> that is
> > >>> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be
> retired
> > >>> >>>>> as LTS and no longer recommended).
> > >>> >>>>>
> > >>> >>>>> I agree that it does not seem there is strong demand for an
> LTS at
> > >>> >>>>> this point. I would propose that we keep 2.16, etc. as
> potential
> > >>> >>>>> candidates, but only declare one as LTS pending demand. The
> question
> > >>> >>>>> of how to keep our tooling stable (or backwards/forwards
> compatible)
> > >>> >>>>> is a good one, especially as we move to drop Python 2.7 in
> 2020 (which
> > >>> >>>>> could itself be a driver for an LTS).
> > >>> >>>>>
> > >>> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <
> kenn@apache.org> wrote:
> > >>> >>>>> >
> > >>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack
> of interest.
> > >>> >>>>> >
> > >>> >>>>> > There are known problems so that I cannot recommend anyone
> to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
> philosophical. I did not like the fact that we had a designated LTS family
> with no usable releases.
> > >>> >>>>> >
> > >>> >>>>> > But many backports were proposed to block 2.7.1 and took a
> very long time to get contirbutors to implement the backports. I ended up
> doing many of them just to move it along. This indicates a lack of interest
> to me. The problem is that we cannot really use a strict cut off date as a
> way to ensure people do the important things and skip the unimportant
> things, because we do know that the issues are critical.
> > >>> >>>>> >
> > >>> >>>>> > And, yes, the fact that Jenkins jobs are separately evolving
> but pretty tightly coupled to the repo contents is a serious problem that I
> wish we had fixed. So verification of each PR was manual.
> > >>> >>>>> >
> > >>> >>>>> > Altogether, I still think LTS is valuable to have as a
> promise to users that we will backport critical fixes. I would like to keep
> that promise and continue to try. Things that are rapidly changing (which
> something always will be) just won't have fixes backported, and that seems
> OK.
> > >>> >>>>> >
> > >>> >>>>> > Kenn
> > >>> >>>>> >
> > >>> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <
> mxm@apache.org> wrote:
> > >>> >>>>> >>
> > >>> >>>>> >> An LTS only makes sense if we end up patching the LTS,
> which so far we
> > >>> >>>>> >> have never done. There has been work done in backporting
> fixes, see
> > >>> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but
> the effort was
> > >>> >>>>> >> never completed. The main reason I believe were
> complications with
> > >>> >>>>> >> running the evolved release scripts against old Beam
> versions.
> > >>> >>>>> >>
> > >>> >>>>> >> Now that the portability layer keeps maturing, it makes me
> optimistic
> > >>> >>>>> >> that we might have a maintained LTS in the future.
> > >>> >>>>> >>
> > >>> >>>>> >> -Max
> > >>> >>>>> >>
> > >>> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
> > >>> >>>>> >> > The fact that end users never asked AFAIK in the ML for
> an LTS and for
> > >>> >>>>> >> > a subsequent minor release of the existing LTS shows IMO
> the low
> > >>> >>>>> >> > interest on having a LTS.
> > >>> >>>>> >> >
> > >>> >>>>> >> > We still are heavily iterating in many areas
> (portability/schema) and
> > >>> >>>>> >> > I am not sure users (and in particular users of open
> source runners)
> > >>> >>>>> >> > get a big benefit of relying on an old version. Maybe
> this is the
> > >>> >>>>> >> > moment to reconsider if having a LTS does even make sense
> given (1)
> > >>> >>>>> >> > that our end user facing APIs are 'mostly' stable (even
> if many still
> > >>> >>>>> >> > called @Experimental). (2) that users get mostly
> improvements on
> > >>> >>>>> >> > runners translation and newer APIs with a low cost just
> by updating
> > >>> >>>>> >> > the version number, and (3) that in case of any
> regression in an
> > >>> >>>>> >> > intermediary release we still can do a minor release even
> if we have
> > >>> >>>>> >> > not yet done so, let's not forget that the only thing we
> need to do
> > >>> >>>>> >> > this is enough interest to do the release from the
> maintainers.
> > >>> >>>>> >> >
> > >>> >>>>> >> >
> > >>> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> > >>> >>>>> >> > <va...@google.com> wrote:
> > >>> >>>>> >> >>
> > >>> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has
> robust Python 3 support compared with prior releases, and also for reasons
> of pending Python 2 deprecation. This has been discussed before [1]. As
> Robert pointed out in that thread, LTS nomination in Beam is currently
> retroactive. If we keep the retroactive policy, the question is how long we
> should wait for a release to be considered "safe" for nomination.  Looks
> like in case of 2.7.0 we waited a month, see [2,3].
> > >>> >>>>> >> >>
> > >>> >>>>> >> >> Thanks,
> > >>> >>>>> >> >> Valentyn
> > >>> >>>>> >> >>
> > >>> >>>>> >> >> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> > >>> >>>>> >> >> [2]
> https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> > >>> >>>>> >> >> [3]
> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
> > >>> >>>>> >> >>
> > >>> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> Hi All,
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> According to our policies page [1]: "There will be at
> least one new LTS release in a 12 month period, and LTS releases are
> considered deprecated after 12 months"
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> Does that mean the next release (2.16) should be the
> next LTS?  It looks like we are in danger of not living up to that promise.
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> Cheers,
> > >>> >>>>> >> >>> Austin
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
> > >>> >>>>> >> >>>
> > >>> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>

Re: Next LTS?

[Ismaël Mejía <ie...@gmail.com>]

+1 to remove any mention of LTS related information to not create
misinformation on users.
Would you be up to do that Austin? or someone else?

On Wed, Mar 25, 2020 at 1:02 AM Robert Bradshaw <ro...@google.com> wrote:
>
> I would want to avoid maintain a Python 2 LTS, even if just for the
> fact that the infrastructure might not be there.
>
> On Tue, Mar 24, 2020 at 3:58 PM Valentyn Tymofieiev <va...@google.com> wrote:
> >
> > Yes, we had a suggestion to pick a stable Python 2 release as an LTS. The suggestion assumed that LTS will continue to exist. Now, if Python 2 is the only reason to have an LTS, we can consider it as long as:
> > - we scope the LTS portion to Python SDK only.
> > - we have an ownership story for Python 2 LTS, for example volunteers in dev or user community who will be willing to maintain that release.
> >
> > We can bring this up when we drop Python 2 support. We decided to revisit that conversation in a couple of months IIRC.
> >
> > On Tue, Mar 24, 2020 at 3:44 PM Ahmet Altay <al...@google.com> wrote:
> >>
> >> Removing it makes sense. We did not have a good way of measuring the demand for LTS releases.
> >>
> >> There was a suggestion to mark the last release with python 2 support to be an LTS release, was there a conclusion on that? ( +Valentyn Tymofieiev )
> >>
> >> Ahmet
> >>
> >> On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com> wrote:
> >>>
> >>> There seems to have been lack of demand. I agree we should remove
> >>> these statements from our site until we find a reason to re-visit
> >>> doing LTS release.
> >>>
> >>> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
> >>> <wh...@gmail.com> wrote:
> >>> >
> >>> > What's our LTS policy these days?  It seems we should remove the following from our site (and encourage GCP does the same, below), if we're not going to maintain these.  I'll update policy page via PR, if get the go ahead that it is our desire.  Seems we can't suggest policies in a policy doc that we don't follow...?
> >>> >
> >>> > I am not trying to suggest demand for LTS.  If others haven't spoken up, that also indicates lack of demand.  Point of my message is to say, we should update our Policies doc, if those aren't what we are practicing (and can re-add later if wanting to revive LTS).
> >>> >
> >>> > https://beam.apache.org/community/policies/
> >>> >
> >>> > Apache Beam aims to make 8 releases in a 12 month period. To accommodate users with longer upgrade cycles, some of these releases will be tagged as long term support (LTS) releases. LTS releases receive patches to fix major issues for 12 months, starting from the release’s initial release date. There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months. The community will mark a release as a LTS release based on various factors, such as the number of LTS releases currently in flight and whether the accumulated feature set since the last LTS provides significant upgrade value. Non-LTS releases do not receive patches and are considered deprecated immediately after the next following minor release. We encourage you to update early and often; do not wait until the deprecation date of the version you are using.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Seems a Google Specific Concern, but related to the community:  https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
> >>> >
> >>> > Apache Beam is an open source, community-led project. Google is part of the community, but we do not own the project or control the release process. We might open bugs or submit patches to the Apache Beam codebase on behalf of Dataflow customers, but we cannot create hotfixes or official releases of Apache Beam on demand.
> >>> >
> >>> > However, the Apache Beam community designates specific releases as long term support (LTS) releases. LTS releases receive patches to fix major issues for a designated period of time. See the Apache Beam policies page for more details about release policies.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:
> >>> >>
> >>> >> I agree with retiring 2.7 as the LTS family. Based on my experience with users 2.7 does not have a particularly high adoption and as pointed out has known critical issues. Declaring another LTS pending demand sounds reasonable but how are we going to gauge this demand?
> >>> >>
> >>> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless we address the tooling problem it seems difficult to feasibly maintain LTS versions over time.
> >>> >>
> >>> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <wh...@gmail.com> wrote:
> >>> >>>
> >>> >>> To be clear, I was picking on - or reminding us of - the promise: I don't have a strong personal need/desire (at least currently) for LTS to exist.  Though, worth ensuring we live up to what we keep on the website.  And, without an active LTS, probably something we should take off the site?
> >>> >>>
> >>> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
> >>> >>>>
> >>> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins jobs without coupling them to the state of the repo during the last Seed Job. It may be that that improvement can help test older LTS-type releases?
> >>> >>>>
> >>> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com> wrote:
> >>> >>>>>
> >>> >>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
> >>> >>>>> we learned some valuable lessons. It would have been good to push out
> >>> >>>>> something (even if it didn't have everything we wanted) but that is
> >>> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
> >>> >>>>> as LTS and no longer recommended).
> >>> >>>>>
> >>> >>>>> I agree that it does not seem there is strong demand for an LTS at
> >>> >>>>> this point. I would propose that we keep 2.16, etc. as potential
> >>> >>>>> candidates, but only declare one as LTS pending demand. The question
> >>> >>>>> of how to keep our tooling stable (or backwards/forwards compatible)
> >>> >>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
> >>> >>>>> could itself be a driver for an LTS).
> >>> >>>>>
> >>> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
> >>> >>>>> >
> >>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of interest.
> >>> >>>>> >
> >>> >>>>> > There are known problems so that I cannot recommend anyone to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was philosophical. I did not like the fact that we had a designated LTS family with no usable releases.
> >>> >>>>> >
> >>> >>>>> > But many backports were proposed to block 2.7.1 and took a very long time to get contirbutors to implement the backports. I ended up doing many of them just to move it along. This indicates a lack of interest to me. The problem is that we cannot really use a strict cut off date as a way to ensure people do the important things and skip the unimportant things, because we do know that the issues are critical.
> >>> >>>>> >
> >>> >>>>> > And, yes, the fact that Jenkins jobs are separately evolving but pretty tightly coupled to the repo contents is a serious problem that I wish we had fixed. So verification of each PR was manual.
> >>> >>>>> >
> >>> >>>>> > Altogether, I still think LTS is valuable to have as a promise to users that we will backport critical fixes. I would like to keep that promise and continue to try. Things that are rapidly changing (which something always will be) just won't have fixes backported, and that seems OK.
> >>> >>>>> >
> >>> >>>>> > Kenn
> >>> >>>>> >
> >>> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org> wrote:
> >>> >>>>> >>
> >>> >>>>> >> An LTS only makes sense if we end up patching the LTS, which so far we
> >>> >>>>> >> have never done. There has been work done in backporting fixes, see
> >>> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
> >>> >>>>> >> never completed. The main reason I believe were complications with
> >>> >>>>> >> running the evolved release scripts against old Beam versions.
> >>> >>>>> >>
> >>> >>>>> >> Now that the portability layer keeps maturing, it makes me optimistic
> >>> >>>>> >> that we might have a maintained LTS in the future.
> >>> >>>>> >>
> >>> >>>>> >> -Max
> >>> >>>>> >>
> >>> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
> >>> >>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and for
> >>> >>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
> >>> >>>>> >> > interest on having a LTS.
> >>> >>>>> >> >
> >>> >>>>> >> > We still are heavily iterating in many areas (portability/schema) and
> >>> >>>>> >> > I am not sure users (and in particular users of open source runners)
> >>> >>>>> >> > get a big benefit of relying on an old version. Maybe this is the
> >>> >>>>> >> > moment to reconsider if having a LTS does even make sense given (1)
> >>> >>>>> >> > that our end user facing APIs are 'mostly' stable (even if many still
> >>> >>>>> >> > called @Experimental). (2) that users get mostly improvements on
> >>> >>>>> >> > runners translation and newer APIs with a low cost just by updating
> >>> >>>>> >> > the version number, and (3) that in case of any regression in an
> >>> >>>>> >> > intermediary release we still can do a minor release even if we have
> >>> >>>>> >> > not yet done so, let's not forget that the only thing we need to do
> >>> >>>>> >> > this is enough interest to do the release from the maintainers.
> >>> >>>>> >> >
> >>> >>>>> >> >
> >>> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> >>> >>>>> >> > <va...@google.com> wrote:
> >>> >>>>> >> >>
> >>> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
> >>> >>>>> >> >>
> >>> >>>>> >> >> Thanks,
> >>> >>>>> >> >> Valentyn
> >>> >>>>> >> >>
> >>> >>>>> >> >> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> >>> >>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> >>> >>>>> >> >> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
> >>> >>>>> >> >>
> >>> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> Hi All,
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> Cheers,
> >>> >>>>> >> >>> Austin
> >>> >>>>> >> >>>
> >>> >>>>> >> >>>
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
> >>> >>>>> >> >>>
> >>> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Robert Bradshaw <ro...@google.com>]

I would want to avoid maintain a Python 2 LTS, even if just for the
fact that the infrastructure might not be there.

On Tue, Mar 24, 2020 at 3:58 PM Valentyn Tymofieiev <va...@google.com> wrote:
>
> Yes, we had a suggestion to pick a stable Python 2 release as an LTS. The suggestion assumed that LTS will continue to exist. Now, if Python 2 is the only reason to have an LTS, we can consider it as long as:
> - we scope the LTS portion to Python SDK only.
> - we have an ownership story for Python 2 LTS, for example volunteers in dev or user community who will be willing to maintain that release.
>
> We can bring this up when we drop Python 2 support. We decided to revisit that conversation in a couple of months IIRC.
>
> On Tue, Mar 24, 2020 at 3:44 PM Ahmet Altay <al...@google.com> wrote:
>>
>> Removing it makes sense. We did not have a good way of measuring the demand for LTS releases.
>>
>> There was a suggestion to mark the last release with python 2 support to be an LTS release, was there a conclusion on that? ( +Valentyn Tymofieiev )
>>
>> Ahmet
>>
>> On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com> wrote:
>>>
>>> There seems to have been lack of demand. I agree we should remove
>>> these statements from our site until we find a reason to re-visit
>>> doing LTS release.
>>>
>>> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
>>> <wh...@gmail.com> wrote:
>>> >
>>> > What's our LTS policy these days?  It seems we should remove the following from our site (and encourage GCP does the same, below), if we're not going to maintain these.  I'll update policy page via PR, if get the go ahead that it is our desire.  Seems we can't suggest policies in a policy doc that we don't follow...?
>>> >
>>> > I am not trying to suggest demand for LTS.  If others haven't spoken up, that also indicates lack of demand.  Point of my message is to say, we should update our Policies doc, if those aren't what we are practicing (and can re-add later if wanting to revive LTS).
>>> >
>>> > https://beam.apache.org/community/policies/
>>> >
>>> > Apache Beam aims to make 8 releases in a 12 month period. To accommodate users with longer upgrade cycles, some of these releases will be tagged as long term support (LTS) releases. LTS releases receive patches to fix major issues for 12 months, starting from the release’s initial release date. There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months. The community will mark a release as a LTS release based on various factors, such as the number of LTS releases currently in flight and whether the accumulated feature set since the last LTS provides significant upgrade value. Non-LTS releases do not receive patches and are considered deprecated immediately after the next following minor release. We encourage you to update early and often; do not wait until the deprecation date of the version you are using.
>>> >
>>> >
>>> >
>>> >
>>> > Seems a Google Specific Concern, but related to the community:  https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
>>> >
>>> > Apache Beam is an open source, community-led project. Google is part of the community, but we do not own the project or control the release process. We might open bugs or submit patches to the Apache Beam codebase on behalf of Dataflow customers, but we cannot create hotfixes or official releases of Apache Beam on demand.
>>> >
>>> > However, the Apache Beam community designates specific releases as long term support (LTS) releases. LTS releases receive patches to fix major issues for a designated period of time. See the Apache Beam policies page for more details about release policies.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:
>>> >>
>>> >> I agree with retiring 2.7 as the LTS family. Based on my experience with users 2.7 does not have a particularly high adoption and as pointed out has known critical issues. Declaring another LTS pending demand sounds reasonable but how are we going to gauge this demand?
>>> >>
>>> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless we address the tooling problem it seems difficult to feasibly maintain LTS versions over time.
>>> >>
>>> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <wh...@gmail.com> wrote:
>>> >>>
>>> >>> To be clear, I was picking on - or reminding us of - the promise: I don't have a strong personal need/desire (at least currently) for LTS to exist.  Though, worth ensuring we live up to what we keep on the website.  And, without an active LTS, probably something we should take off the site?
>>> >>>
>>> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
>>> >>>>
>>> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins jobs without coupling them to the state of the repo during the last Seed Job. It may be that that improvement can help test older LTS-type releases?
>>> >>>>
>>> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com> wrote:
>>> >>>>>
>>> >>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>> >>>>> we learned some valuable lessons. It would have been good to push out
>>> >>>>> something (even if it didn't have everything we wanted) but that is
>>> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>> >>>>> as LTS and no longer recommended).
>>> >>>>>
>>> >>>>> I agree that it does not seem there is strong demand for an LTS at
>>> >>>>> this point. I would propose that we keep 2.16, etc. as potential
>>> >>>>> candidates, but only declare one as LTS pending demand. The question
>>> >>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>> >>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>> >>>>> could itself be a driver for an LTS).
>>> >>>>>
>>> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
>>> >>>>> >
>>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of interest.
>>> >>>>> >
>>> >>>>> > There are known problems so that I cannot recommend anyone to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was philosophical. I did not like the fact that we had a designated LTS family with no usable releases.
>>> >>>>> >
>>> >>>>> > But many backports were proposed to block 2.7.1 and took a very long time to get contirbutors to implement the backports. I ended up doing many of them just to move it along. This indicates a lack of interest to me. The problem is that we cannot really use a strict cut off date as a way to ensure people do the important things and skip the unimportant things, because we do know that the issues are critical.
>>> >>>>> >
>>> >>>>> > And, yes, the fact that Jenkins jobs are separately evolving but pretty tightly coupled to the repo contents is a serious problem that I wish we had fixed. So verification of each PR was manual.
>>> >>>>> >
>>> >>>>> > Altogether, I still think LTS is valuable to have as a promise to users that we will backport critical fixes. I would like to keep that promise and continue to try. Things that are rapidly changing (which something always will be) just won't have fixes backported, and that seems OK.
>>> >>>>> >
>>> >>>>> > Kenn
>>> >>>>> >
>>> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org> wrote:
>>> >>>>> >>
>>> >>>>> >> An LTS only makes sense if we end up patching the LTS, which so far we
>>> >>>>> >> have never done. There has been work done in backporting fixes, see
>>> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
>>> >>>>> >> never completed. The main reason I believe were complications with
>>> >>>>> >> running the evolved release scripts against old Beam versions.
>>> >>>>> >>
>>> >>>>> >> Now that the portability layer keeps maturing, it makes me optimistic
>>> >>>>> >> that we might have a maintained LTS in the future.
>>> >>>>> >>
>>> >>>>> >> -Max
>>> >>>>> >>
>>> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>> >>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and for
>>> >>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>> >>>>> >> > interest on having a LTS.
>>> >>>>> >> >
>>> >>>>> >> > We still are heavily iterating in many areas (portability/schema) and
>>> >>>>> >> > I am not sure users (and in particular users of open source runners)
>>> >>>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>> >>>>> >> > moment to reconsider if having a LTS does even make sense given (1)
>>> >>>>> >> > that our end user facing APIs are 'mostly' stable (even if many still
>>> >>>>> >> > called @Experimental). (2) that users get mostly improvements on
>>> >>>>> >> > runners translation and newer APIs with a low cost just by updating
>>> >>>>> >> > the version number, and (3) that in case of any regression in an
>>> >>>>> >> > intermediary release we still can do a minor release even if we have
>>> >>>>> >> > not yet done so, let's not forget that the only thing we need to do
>>> >>>>> >> > this is enough interest to do the release from the maintainers.
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>> >>>>> >> > <va...@google.com> wrote:
>>> >>>>> >> >>
>>> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
>>> >>>>> >> >>
>>> >>>>> >> >> Thanks,
>>> >>>>> >> >> Valentyn
>>> >>>>> >> >>
>>> >>>>> >> >> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> >>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>> >>>>> >> >> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>> >>>>> >> >>
>>> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
>>> >>>>> >> >>>
>>> >>>>> >> >>> Hi All,
>>> >>>>> >> >>>
>>> >>>>> >> >>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
>>> >>>>> >> >>>
>>> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
>>> >>>>> >> >>>
>>> >>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
>>> >>>>> >> >>>
>>> >>>>> >> >>> Cheers,
>>> >>>>> >> >>> Austin
>>> >>>>> >> >>>
>>> >>>>> >> >>>
>>> >>>>> >> >>>
>>> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
>>> >>>>> >> >>>
>>> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Valentyn Tymofieiev <va...@google.com>]

Yes, we had a suggestion to pick a stable Python 2 release as an LTS. The
suggestion assumed that LTS will continue to exist. Now, if Python 2 is
the only reason to have an LTS, we can consider it as long as:
- we scope the LTS portion to Python SDK only.
- we have an ownership story for Python 2 LTS, for example volunteers in
dev or user community who will be willing to maintain that release.

We can bring this up when we drop Python 2 support. We decided to revisit
that conversation in a couple of months IIRC.

On Tue, Mar 24, 2020 at 3:44 PM Ahmet Altay <al...@google.com> wrote:

> Removing it makes sense. We did not have a good way of measuring the
> demand for LTS releases.
>
> There was a suggestion to mark the last release with python 2 support to
> be an LTS release, was there a conclusion on that? ( +Valentyn Tymofieiev
> <va...@google.com> )
>
> Ahmet
>
> On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> There seems to have been lack of demand. I agree we should remove
>> these statements from our site until we find a reason to re-visit
>> doing LTS release.
>>
>> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
>> <wh...@gmail.com> wrote:
>> >
>> > What's our LTS policy these days?  It seems we should remove the
>> following from our site (and encourage GCP does the same, below), if we're
>> not going to maintain these.  I'll update policy page via PR, if get the go
>> ahead that it is our desire.  Seems we can't suggest policies in a policy
>> doc that we don't follow...?
>> >
>> > I am not trying to suggest demand for LTS.  If others haven't spoken
>> up, that also indicates lack of demand.  Point of my message is to say, we
>> should update our Policies doc, if those aren't what we are practicing (and
>> can re-add later if wanting to revive LTS).
>> >
>> > https://beam.apache.org/community/policies/
>> >
>> > Apache Beam aims to make 8 releases in a 12 month period. To
>> accommodate users with longer upgrade cycles, some of these releases will
>> be tagged as long term support (LTS) releases. LTS releases receive patches
>> to fix major issues for 12 months, starting from the release’s initial
>> release date. There will be at least one new LTS release in a 12 month
>> period, and LTS releases are considered deprecated after 12 months. The
>> community will mark a release as a LTS release based on various factors,
>> such as the number of LTS releases currently in flight and whether the
>> accumulated feature set since the last LTS provides significant upgrade
>> value. Non-LTS releases do not receive patches and are considered
>> deprecated immediately after the next following minor release. We encourage
>> you to update early and often; do not wait until the deprecation date of
>> the version you are using.
>> >
>> >
>> >
>> >
>> > Seems a Google Specific Concern, but related to the community:
>> https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
>> >
>> > Apache Beam is an open source, community-led project. Google is part of
>> the community, but we do not own the project or control the release
>> process. We might open bugs or submit patches to the Apache Beam codebase
>> on behalf of Dataflow customers, but we cannot create hotfixes or official
>> releases of Apache Beam on demand.
>> >
>> > However, the Apache Beam community designates specific releases as long
>> term support (LTS) releases. LTS releases receive patches to fix major
>> issues for a designated period of time. See the Apache Beam policies page
>> for more details about release policies.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> I agree with retiring 2.7 as the LTS family. Based on my experience
>> with users 2.7 does not have a particularly high adoption and as pointed
>> out has known critical issues. Declaring another LTS pending demand sounds
>> reasonable but how are we going to gauge this demand?
>> >>
>> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless we
>> address the tooling problem it seems difficult to feasibly maintain LTS
>> versions over time.
>> >>
>> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>> >>>
>> >>> To be clear, I was picking on - or reminding us of - the promise: I
>> don't have a strong personal need/desire (at least currently) for LTS to
>> exist.  Though, worth ensuring we live up to what we keep on the website.
>> And, without an active LTS, probably something we should take off the site?
>> >>>
>> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com>
>> wrote:
>> >>>>
>> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins jobs
>> without coupling them to the state of the repo during the last Seed Job. It
>> may be that that improvement can help test older LTS-type releases?
>> >>>>
>> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>>>>
>> >>>>> In many ways the 2.7 LTS was trying to flesh out the process. I
>> think
>> >>>>> we learned some valuable lessons. It would have been good to push
>> out
>> >>>>> something (even if it didn't have everything we wanted) but that is
>> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be
>> retired
>> >>>>> as LTS and no longer recommended).
>> >>>>>
>> >>>>> I agree that it does not seem there is strong demand for an LTS at
>> >>>>> this point. I would propose that we keep 2.16, etc. as potential
>> >>>>> candidates, but only declare one as LTS pending demand. The question
>> >>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>> >>>>> is a good one, especially as we move to drop Python 2.7 in 2020
>> (which
>> >>>>> could itself be a driver for an LTS).
>> >>>>>
>> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
>> wrote:
>> >>>>> >
>> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>> interest.
>> >>>>> >
>> >>>>> > There are known problems so that I cannot recommend anyone to use
>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>> philosophical. I did not like the fact that we had a designated LTS family
>> with no usable releases.
>> >>>>> >
>> >>>>> > But many backports were proposed to block 2.7.1 and took a very
>> long time to get contirbutors to implement the backports. I ended up doing
>> many of them just to move it along. This indicates a lack of interest to
>> me. The problem is that we cannot really use a strict cut off date as a way
>> to ensure people do the important things and skip the unimportant things,
>> because we do know that the issues are critical.
>> >>>>> >
>> >>>>> > And, yes, the fact that Jenkins jobs are separately evolving but
>> pretty tightly coupled to the repo contents is a serious problem that I
>> wish we had fixed. So verification of each PR was manual.
>> >>>>> >
>> >>>>> > Altogether, I still think LTS is valuable to have as a promise to
>> users that we will backport critical fixes. I would like to keep that
>> promise and continue to try. Things that are rapidly changing (which
>> something always will be) just won't have fixes backported, and that seems
>> OK.
>> >>>>> >
>> >>>>> > Kenn
>> >>>>> >
>> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <
>> mxm@apache.org> wrote:
>> >>>>> >>
>> >>>>> >> An LTS only makes sense if we end up patching the LTS, which so
>> far we
>> >>>>> >> have never done. There has been work done in backporting fixes,
>> see
>> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the
>> effort was
>> >>>>> >> never completed. The main reason I believe were complications
>> with
>> >>>>> >> running the evolved release scripts against old Beam versions.
>> >>>>> >>
>> >>>>> >> Now that the portability layer keeps maturing, it makes me
>> optimistic
>> >>>>> >> that we might have a maintained LTS in the future.
>> >>>>> >>
>> >>>>> >> -Max
>> >>>>> >>
>> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>> >>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS
>> and for
>> >>>>> >> > a subsequent minor release of the existing LTS shows IMO the
>> low
>> >>>>> >> > interest on having a LTS.
>> >>>>> >> >
>> >>>>> >> > We still are heavily iterating in many areas
>> (portability/schema) and
>> >>>>> >> > I am not sure users (and in particular users of open source
>> runners)
>> >>>>> >> > get a big benefit of relying on an old version. Maybe this is
>> the
>> >>>>> >> > moment to reconsider if having a LTS does even make sense
>> given (1)
>> >>>>> >> > that our end user facing APIs are 'mostly' stable (even if
>> many still
>> >>>>> >> > called @Experimental). (2) that users get mostly improvements
>> on
>> >>>>> >> > runners translation and newer APIs with a low cost just by
>> updating
>> >>>>> >> > the version number, and (3) that in case of any regression in
>> an
>> >>>>> >> > intermediary release we still can do a minor release even if
>> we have
>> >>>>> >> > not yet done so, let's not forget that the only thing we need
>> to do
>> >>>>> >> > this is enough interest to do the release from the maintainers.
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>> >>>>> >> > <va...@google.com> wrote:
>> >>>>> >> >>
>> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has
>> robust Python 3 support compared with prior releases, and also for reasons
>> of pending Python 2 deprecation. This has been discussed before [1]. As
>> Robert pointed out in that thread, LTS nomination in Beam is currently
>> retroactive. If we keep the retroactive policy, the question is how long we
>> should wait for a release to be considered "safe" for nomination.  Looks
>> like in case of 2.7.0 we waited a month, see [2,3].
>> >>>>> >> >>
>> >>>>> >> >> Thanks,
>> >>>>> >> >> Valentyn
>> >>>>> >> >>
>> >>>>> >> >> [1]
>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> >>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>> >>>>> >> >> [3]
>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>> >>>>> >> >>
>> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>> >>>>> >> >>>
>> >>>>> >> >>> Hi All,
>> >>>>> >> >>>
>> >>>>> >> >>> According to our policies page [1]: "There will be at least
>> one new LTS release in a 12 month period, and LTS releases are considered
>> deprecated after 12 months"
>> >>>>> >> >>>
>> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
>> >>>>> >> >>>
>> >>>>> >> >>> Does that mean the next release (2.16) should be the next
>> LTS?  It looks like we are in danger of not living up to that promise.
>> >>>>> >> >>>
>> >>>>> >> >>> Cheers,
>> >>>>> >> >>> Austin
>> >>>>> >> >>>
>> >>>>> >> >>>
>> >>>>> >> >>>
>> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
>> >>>>> >> >>>
>> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>
>

Re: Next LTS?

[Ahmet Altay <al...@google.com>]

Removing it makes sense. We did not have a good way of measuring the demand
for LTS releases.

There was a suggestion to mark the last release with python 2 support to be
an LTS release, was there a conclusion on that? ( +Valentyn Tymofieiev
<va...@google.com> )

Ahmet

On Tue, Mar 24, 2020 at 2:34 PM Robert Bradshaw <ro...@google.com> wrote:

> There seems to have been lack of demand. I agree we should remove
> these statements from our site until we find a reason to re-visit
> doing LTS release.
>
> On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
> <wh...@gmail.com> wrote:
> >
> > What's our LTS policy these days?  It seems we should remove the
> following from our site (and encourage GCP does the same, below), if we're
> not going to maintain these.  I'll update policy page via PR, if get the go
> ahead that it is our desire.  Seems we can't suggest policies in a policy
> doc that we don't follow...?
> >
> > I am not trying to suggest demand for LTS.  If others haven't spoken up,
> that also indicates lack of demand.  Point of my message is to say, we
> should update our Policies doc, if those aren't what we are practicing (and
> can re-add later if wanting to revive LTS).
> >
> > https://beam.apache.org/community/policies/
> >
> > Apache Beam aims to make 8 releases in a 12 month period. To accommodate
> users with longer upgrade cycles, some of these releases will be tagged as
> long term support (LTS) releases. LTS releases receive patches to fix major
> issues for 12 months, starting from the release’s initial release date.
> There will be at least one new LTS release in a 12 month period, and LTS
> releases are considered deprecated after 12 months. The community will mark
> a release as a LTS release based on various factors, such as the number of
> LTS releases currently in flight and whether the accumulated feature set
> since the last LTS provides significant upgrade value. Non-LTS releases do
> not receive patches and are considered deprecated immediately after the
> next following minor release. We encourage you to update early and often;
> do not wait until the deprecation date of the version you are using.
> >
> >
> >
> >
> > Seems a Google Specific Concern, but related to the community:
> https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
> >
> > Apache Beam is an open source, community-led project. Google is part of
> the community, but we do not own the project or control the release
> process. We might open bugs or submit patches to the Apache Beam codebase
> on behalf of Dataflow customers, but we cannot create hotfixes or official
> releases of Apache Beam on demand.
> >
> > However, the Apache Beam community designates specific releases as long
> term support (LTS) releases. LTS releases receive patches to fix major
> issues for a designated period of time. See the Apache Beam policies page
> for more details about release policies.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:
> >>
> >> I agree with retiring 2.7 as the LTS family. Based on my experience
> with users 2.7 does not have a particularly high adoption and as pointed
> out has known critical issues. Declaring another LTS pending demand sounds
> reasonable but how are we going to gauge this demand?
> >>
> >> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless we
> address the tooling problem it seems difficult to feasibly maintain LTS
> versions over time.
> >>
> >> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> >>>
> >>> To be clear, I was picking on - or reminding us of - the promise: I
> don't have a strong personal need/desire (at least currently) for LTS to
> exist.  Though, worth ensuring we live up to what we keep on the website.
> And, without an active LTS, probably something we should take off the site?
> >>>
> >>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>>>
> >>>> +Łukasz Gajowy had at some point thought of setting up jenkins jobs
> without coupling them to the state of the repo during the last Seed Job. It
> may be that that improvement can help test older LTS-type releases?
> >>>>
> >>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>>>
> >>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
> >>>>> we learned some valuable lessons. It would have been good to push out
> >>>>> something (even if it didn't have everything we wanted) but that is
> >>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
> >>>>> as LTS and no longer recommended).
> >>>>>
> >>>>> I agree that it does not seem there is strong demand for an LTS at
> >>>>> this point. I would propose that we keep 2.16, etc. as potential
> >>>>> candidates, but only declare one as LTS pending demand. The question
> >>>>> of how to keep our tooling stable (or backwards/forwards compatible)
> >>>>> is a good one, especially as we move to drop Python 2.7 in 2020
> (which
> >>>>> could itself be a driver for an LTS).
> >>>>>
> >>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
> wrote:
> >>>>> >
> >>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
> interest.
> >>>>> >
> >>>>> > There are known problems so that I cannot recommend anyone to use
> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
> philosophical. I did not like the fact that we had a designated LTS family
> with no usable releases.
> >>>>> >
> >>>>> > But many backports were proposed to block 2.7.1 and took a very
> long time to get contirbutors to implement the backports. I ended up doing
> many of them just to move it along. This indicates a lack of interest to
> me. The problem is that we cannot really use a strict cut off date as a way
> to ensure people do the important things and skip the unimportant things,
> because we do know that the issues are critical.
> >>>>> >
> >>>>> > And, yes, the fact that Jenkins jobs are separately evolving but
> pretty tightly coupled to the repo contents is a serious problem that I
> wish we had fixed. So verification of each PR was manual.
> >>>>> >
> >>>>> > Altogether, I still think LTS is valuable to have as a promise to
> users that we will backport critical fixes. I would like to keep that
> promise and continue to try. Things that are rapidly changing (which
> something always will be) just won't have fixes backported, and that seems
> OK.
> >>>>> >
> >>>>> > Kenn
> >>>>> >
> >>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <
> mxm@apache.org> wrote:
> >>>>> >>
> >>>>> >> An LTS only makes sense if we end up patching the LTS, which so
> far we
> >>>>> >> have never done. There has been work done in backporting fixes,
> see
> >>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the
> effort was
> >>>>> >> never completed. The main reason I believe were complications with
> >>>>> >> running the evolved release scripts against old Beam versions.
> >>>>> >>
> >>>>> >> Now that the portability layer keeps maturing, it makes me
> optimistic
> >>>>> >> that we might have a maintained LTS in the future.
> >>>>> >>
> >>>>> >> -Max
> >>>>> >>
> >>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
> >>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS
> and for
> >>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
> >>>>> >> > interest on having a LTS.
> >>>>> >> >
> >>>>> >> > We still are heavily iterating in many areas
> (portability/schema) and
> >>>>> >> > I am not sure users (and in particular users of open source
> runners)
> >>>>> >> > get a big benefit of relying on an old version. Maybe this is
> the
> >>>>> >> > moment to reconsider if having a LTS does even make sense given
> (1)
> >>>>> >> > that our end user facing APIs are 'mostly' stable (even if many
> still
> >>>>> >> > called @Experimental). (2) that users get mostly improvements on
> >>>>> >> > runners translation and newer APIs with a low cost just by
> updating
> >>>>> >> > the version number, and (3) that in case of any regression in an
> >>>>> >> > intermediary release we still can do a minor release even if we
> have
> >>>>> >> > not yet done so, let's not forget that the only thing we need
> to do
> >>>>> >> > this is enough interest to do the release from the maintainers.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> >>>>> >> > <va...@google.com> wrote:
> >>>>> >> >>
> >>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust
> Python 3 support compared with prior releases, and also for reasons of
> pending Python 2 deprecation. This has been discussed before [1]. As Robert
> pointed out in that thread, LTS nomination in Beam is currently
> retroactive. If we keep the retroactive policy, the question is how long we
> should wait for a release to be considered "safe" for nomination.  Looks
> like in case of 2.7.0 we waited a month, see [2,3].
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >> Valentyn
> >>>>> >> >>
> >>>>> >> >> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> >>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> >>>>> >> >> [3]
> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
> >>>>> >> >>
> >>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> >>>>> >> >>>
> >>>>> >> >>> Hi All,
> >>>>> >> >>>
> >>>>> >> >>> According to our policies page [1]: "There will be at least
> one new LTS release in a 12 month period, and LTS releases are considered
> deprecated after 12 months"
> >>>>> >> >>>
> >>>>> >> >>> The last LTS was released 2018-10-02 [2].
> >>>>> >> >>>
> >>>>> >> >>> Does that mean the next release (2.16) should be the next
> LTS?  It looks like we are in danger of not living up to that promise.
> >>>>> >> >>>
> >>>>> >> >>> Cheers,
> >>>>> >> >>> Austin
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>>
> >>>>> >> >>> [1] https://beam.apache.org/community/policies/
> >>>>> >> >>>
> >>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>

Re: Next LTS?

[Robert Bradshaw <ro...@google.com>]

There seems to have been lack of demand. I agree we should remove
these statements from our site until we find a reason to re-visit
doing LTS release.

On Tue, Mar 24, 2020 at 2:23 PM Austin Bennett
<wh...@gmail.com> wrote:
>
> What's our LTS policy these days?  It seems we should remove the following from our site (and encourage GCP does the same, below), if we're not going to maintain these.  I'll update policy page via PR, if get the go ahead that it is our desire.  Seems we can't suggest policies in a policy doc that we don't follow...?
>
> I am not trying to suggest demand for LTS.  If others haven't spoken up, that also indicates lack of demand.  Point of my message is to say, we should update our Policies doc, if those aren't what we are practicing (and can re-add later if wanting to revive LTS).
>
> https://beam.apache.org/community/policies/
>
> Apache Beam aims to make 8 releases in a 12 month period. To accommodate users with longer upgrade cycles, some of these releases will be tagged as long term support (LTS) releases. LTS releases receive patches to fix major issues for 12 months, starting from the release’s initial release date. There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months. The community will mark a release as a LTS release based on various factors, such as the number of LTS releases currently in flight and whether the accumulated feature set since the last LTS provides significant upgrade value. Non-LTS releases do not receive patches and are considered deprecated immediately after the next following minor release. We encourage you to update early and often; do not wait until the deprecation date of the version you are using.
>
>
>
>
> Seems a Google Specific Concern, but related to the community:  https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x
>
> Apache Beam is an open source, community-led project. Google is part of the community, but we do not own the project or control the release process. We might open bugs or submit patches to the Apache Beam codebase on behalf of Dataflow customers, but we cannot create hotfixes or official releases of Apache Beam on demand.
>
> However, the Apache Beam community designates specific releases as long term support (LTS) releases. LTS releases receive patches to fix major issues for a designated period of time. See the Apache Beam policies page for more details about release policies.
>
>
>
>
>
>
>
>
>
> On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:
>>
>> I agree with retiring 2.7 as the LTS family. Based on my experience with users 2.7 does not have a particularly high adoption and as pointed out has known critical issues. Declaring another LTS pending demand sounds reasonable but how are we going to gauge this demand?
>>
>> +Yifan Zou +Alan Myrvold on the tooling question as well. Unless we address the tooling problem it seems difficult to feasibly maintain LTS versions over time.
>>
>> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <wh...@gmail.com> wrote:
>>>
>>> To be clear, I was picking on - or reminding us of - the promise: I don't have a strong personal need/desire (at least currently) for LTS to exist.  Though, worth ensuring we live up to what we keep on the website.  And, without an active LTS, probably something we should take off the site?
>>>
>>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
>>>>
>>>> +Łukasz Gajowy had at some point thought of setting up jenkins jobs without coupling them to the state of the repo during the last Seed Job. It may be that that improvement can help test older LTS-type releases?
>>>>
>>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>
>>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>>>> we learned some valuable lessons. It would have been good to push out
>>>>> something (even if it didn't have everything we wanted) but that is
>>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>>>> as LTS and no longer recommended).
>>>>>
>>>>> I agree that it does not seem there is strong demand for an LTS at
>>>>> this point. I would propose that we keep 2.16, etc. as potential
>>>>> candidates, but only declare one as LTS pending demand. The question
>>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>>>> could itself be a driver for an LTS).
>>>>>
>>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>>> >
>>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of interest.
>>>>> >
>>>>> > There are known problems so that I cannot recommend anyone to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was philosophical. I did not like the fact that we had a designated LTS family with no usable releases.
>>>>> >
>>>>> > But many backports were proposed to block 2.7.1 and took a very long time to get contirbutors to implement the backports. I ended up doing many of them just to move it along. This indicates a lack of interest to me. The problem is that we cannot really use a strict cut off date as a way to ensure people do the important things and skip the unimportant things, because we do know that the issues are critical.
>>>>> >
>>>>> > And, yes, the fact that Jenkins jobs are separately evolving but pretty tightly coupled to the repo contents is a serious problem that I wish we had fixed. So verification of each PR was manual.
>>>>> >
>>>>> > Altogether, I still think LTS is valuable to have as a promise to users that we will backport critical fixes. I would like to keep that promise and continue to try. Things that are rapidly changing (which something always will be) just won't have fixes backported, and that seems OK.
>>>>> >
>>>>> > Kenn
>>>>> >
>>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org> wrote:
>>>>> >>
>>>>> >> An LTS only makes sense if we end up patching the LTS, which so far we
>>>>> >> have never done. There has been work done in backporting fixes, see
>>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
>>>>> >> never completed. The main reason I believe were complications with
>>>>> >> running the evolved release scripts against old Beam versions.
>>>>> >>
>>>>> >> Now that the portability layer keeps maturing, it makes me optimistic
>>>>> >> that we might have a maintained LTS in the future.
>>>>> >>
>>>>> >> -Max
>>>>> >>
>>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and for
>>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>>>> >> > interest on having a LTS.
>>>>> >> >
>>>>> >> > We still are heavily iterating in many areas (portability/schema) and
>>>>> >> > I am not sure users (and in particular users of open source runners)
>>>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>>>> >> > moment to reconsider if having a LTS does even make sense given (1)
>>>>> >> > that our end user facing APIs are 'mostly' stable (even if many still
>>>>> >> > called @Experimental). (2) that users get mostly improvements on
>>>>> >> > runners translation and newer APIs with a low cost just by updating
>>>>> >> > the version number, and (3) that in case of any regression in an
>>>>> >> > intermediary release we still can do a minor release even if we have
>>>>> >> > not yet done so, let's not forget that the only thing we need to do
>>>>> >> > this is enough interest to do the release from the maintainers.
>>>>> >> >
>>>>> >> >
>>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>>>> >> > <va...@google.com> wrote:
>>>>> >> >>
>>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >> Valentyn
>>>>> >> >>
>>>>> >> >> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>>>> >> >> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>>>> >> >>
>>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
>>>>> >> >>>
>>>>> >> >>> Hi All,
>>>>> >> >>>
>>>>> >> >>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
>>>>> >> >>>
>>>>> >> >>> The last LTS was released 2018-10-02 [2].
>>>>> >> >>>
>>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
>>>>> >> >>>
>>>>> >> >>> Cheers,
>>>>> >> >>> Austin
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>>
>>>>> >> >>> [1] https://beam.apache.org/community/policies/
>>>>> >> >>>
>>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Austin Bennett <wh...@gmail.com>]

What's our LTS policy these days?  It seems we should remove the following
from our site (and encourage GCP does the same, below), if we're not going
to maintain these.  I'll update policy page via PR, if get the go ahead
that it is our desire.  Seems we can't suggest policies in a policy doc
that we don't follow...?

I am not trying to suggest demand for LTS.  If others haven't spoken up,
that also indicates lack of demand.  Point of my message is to say, we
should update our Policies doc, if those aren't what we are practicing (and
can re-add later if wanting to revive LTS).

https://beam.apache.org/community/policies/

Apache Beam aims to make 8 releases in a 12 month period. To accommodate
users with longer upgrade cycles, some of these releases will be tagged as
long term support (LTS) releases. LTS releases receive patches to fix major
issues for 12 months, starting from the release’s initial release date.
There will be at least one new LTS release in a 12 month period, and LTS
releases are considered deprecated after 12 months. The community will mark
a release as a LTS release based on various factors, such as the number of
LTS releases currently in flight and whether the accumulated feature set
since the last LTS provides significant upgrade value. Non-LTS releases do
not receive patches and are considered deprecated immediately after the
next following minor release. We encourage you to update early and often;
do not wait until the deprecation date of the version you are using.



Seems a Google Specific Concern, but related to the community:
https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#apache-beam-sdks-2x

Apache Beam <http://beam.apache.org/> is an open source, community-led
project. Google is part of the community, but we do not own the project or
control the release process. We might open bugs or submit patches to the
Apache Beam codebase on behalf of Dataflow customers, but we cannot create
hotfixes or official releases of Apache Beam on demand.

However, the Apache Beam community designates specific releases as *long
term support (LTS)* releases. LTS releases receive patches to fix major
issues for a designated period of time. See the Apache Beam policies
<https://beam.apache.org/community/policies/> page for more details about
release policies.









On Thu, Sep 19, 2019 at 5:01 PM Ahmet Altay <al...@google.com> wrote:

> I agree with retiring 2.7 as the LTS family. Based on my experience with
> users 2.7 does not have a particularly high adoption and as pointed out has
> known critical issues. Declaring another LTS pending demand sounds
> reasonable but how are we going to gauge this demand?
>
> +Yifan Zou <yi...@google.com> +Alan Myrvold <am...@google.com> on
> the tooling question as well. Unless we address the tooling problem it
> seems difficult to feasibly maintain LTS versions over time.
>
> On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
>
>> To be clear, I was picking on - or reminding us of - the promise: I don't
>> have a strong personal need/desire (at least currently) for LTS to exist.
>> Though, worth ensuring we live up to what we keep on the website.  And,
>> without an active LTS, probably something we should take off the site?
>>
>> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> +Łukasz Gajowy <lu...@polidea.com> had at some point thought of
>>> setting up jenkins jobs without coupling them to the state of the repo
>>> during the last Seed Job. It may be that that improvement can help test
>>> older LTS-type releases?
>>>
>>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>>> we learned some valuable lessons. It would have been good to push out
>>>> something (even if it didn't have everything we wanted) but that is
>>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>>> as LTS and no longer recommended).
>>>>
>>>> I agree that it does not seem there is strong demand for an LTS at
>>>> this point. I would propose that we keep 2.16, etc. as potential
>>>> candidates, but only declare one as LTS pending demand. The question
>>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>>> could itself be a driver for an LTS).
>>>>
>>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
>>>> wrote:
>>>> >
>>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>>>> interest.
>>>> >
>>>> > There are known problems so that I cannot recommend anyone to use
>>>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>>>> philosophical. I did not like the fact that we had a designated LTS family
>>>> with no usable releases.
>>>> >
>>>> > But many backports were proposed to block 2.7.1 and took a very long
>>>> time to get contirbutors to implement the backports. I ended up doing many
>>>> of them just to move it along. This indicates a lack of interest to me. The
>>>> problem is that we cannot really use a strict cut off date as a way to
>>>> ensure people do the important things and skip the unimportant things,
>>>> because we do know that the issues are critical.
>>>> >
>>>> > And, yes, the fact that Jenkins jobs are separately evolving but
>>>> pretty tightly coupled to the repo contents is a serious problem that I
>>>> wish we had fixed. So verification of each PR was manual.
>>>> >
>>>> > Altogether, I still think LTS is valuable to have as a promise to
>>>> users that we will backport critical fixes. I would like to keep that
>>>> promise and continue to try. Things that are rapidly changing (which
>>>> something always will be) just won't have fixes backported, and that seems
>>>> OK.
>>>> >
>>>> > Kenn
>>>> >
>>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> An LTS only makes sense if we end up patching the LTS, which so far
>>>> we
>>>> >> have never done. There has been work done in backporting fixes, see
>>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort
>>>> was
>>>> >> never completed. The main reason I believe were complications with
>>>> >> running the evolved release scripts against old Beam versions.
>>>> >>
>>>> >> Now that the portability layer keeps maturing, it makes me optimistic
>>>> >> that we might have a maintained LTS in the future.
>>>> >>
>>>> >> -Max
>>>> >>
>>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and
>>>> for
>>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>>> >> > interest on having a LTS.
>>>> >> >
>>>> >> > We still are heavily iterating in many areas (portability/schema)
>>>> and
>>>> >> > I am not sure users (and in particular users of open source
>>>> runners)
>>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>>> >> > moment to reconsider if having a LTS does even make sense given (1)
>>>> >> > that our end user facing APIs are 'mostly' stable (even if many
>>>> still
>>>> >> > called @Experimental). (2) that users get mostly improvements on
>>>> >> > runners translation and newer APIs with a low cost just by updating
>>>> >> > the version number, and (3) that in case of any regression in an
>>>> >> > intermediary release we still can do a minor release even if we
>>>> have
>>>> >> > not yet done so, let's not forget that the only thing we need to do
>>>> >> > this is enough interest to do the release from the maintainers.
>>>> >> >
>>>> >> >
>>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>>> >> > <va...@google.com> wrote:
>>>> >> >>
>>>> >> >> I support nominating 2.16.0 as LTS release since in has robust
>>>> Python 3 support compared with prior releases, and also for reasons of
>>>> pending Python 2 deprecation. This has been discussed before [1]. As Robert
>>>> pointed out in that thread, LTS nomination in Beam is currently
>>>> retroactive. If we keep the retroactive policy, the question is how long we
>>>> should wait for a release to be considered "safe" for nomination.  Looks
>>>> like in case of 2.7.0 we waited a month, see [2,3].
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Valentyn
>>>> >> >>
>>>> >> >> [1]
>>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>>> >> >> [3]
>>>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>>> >> >>
>>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>>>> whatwouldaustindo@gmail.com> wrote:
>>>> >> >>>
>>>> >> >>> Hi All,
>>>> >> >>>
>>>> >> >>> According to our policies page [1]: "There will be at least one
>>>> new LTS release in a 12 month period, and LTS releases are considered
>>>> deprecated after 12 months"
>>>> >> >>>
>>>> >> >>> The last LTS was released 2018-10-02 [2].
>>>> >> >>>
>>>> >> >>> Does that mean the next release (2.16) should be the next LTS?
>>>> It looks like we are in danger of not living up to that promise.
>>>> >> >>>
>>>> >> >>> Cheers,
>>>> >> >>> Austin
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> [1] https://beam.apache.org/community/policies/
>>>> >> >>>
>>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>>>
>>>

Re: Next LTS?

[Ahmet Altay <al...@google.com>]

I agree with retiring 2.7 as the LTS family. Based on my experience with
users 2.7 does not have a particularly high adoption and as pointed out has
known critical issues. Declaring another LTS pending demand sounds
reasonable but how are we going to gauge this demand?

+Yifan Zou <yi...@google.com> +Alan Myrvold <am...@google.com> on the
tooling question as well. Unless we address the tooling problem it seems
difficult to feasibly maintain LTS versions over time.

On Thu, Sep 19, 2019 at 3:45 PM Austin Bennett <wh...@gmail.com>
wrote:

> To be clear, I was picking on - or reminding us of - the promise: I don't
> have a strong personal need/desire (at least currently) for LTS to exist.
> Though, worth ensuring we live up to what we keep on the website.  And,
> without an active LTS, probably something we should take off the site?
>
> On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:
>
>> +Łukasz Gajowy <lu...@polidea.com> had at some point thought of
>> setting up jenkins jobs without coupling them to the state of the repo
>> during the last Seed Job. It may be that that improvement can help test
>> older LTS-type releases?
>>
>> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>>> we learned some valuable lessons. It would have been good to push out
>>> something (even if it didn't have everything we wanted) but that is
>>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>>> as LTS and no longer recommended).
>>>
>>> I agree that it does not seem there is strong demand for an LTS at
>>> this point. I would propose that we keep 2.16, etc. as potential
>>> candidates, but only declare one as LTS pending demand. The question
>>> of how to keep our tooling stable (or backwards/forwards compatible)
>>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>>> could itself be a driver for an LTS).
>>>
>>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org>
>>> wrote:
>>> >
>>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>>> interest.
>>> >
>>> > There are known problems so that I cannot recommend anyone to use
>>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>>> philosophical. I did not like the fact that we had a designated LTS family
>>> with no usable releases.
>>> >
>>> > But many backports were proposed to block 2.7.1 and took a very long
>>> time to get contirbutors to implement the backports. I ended up doing many
>>> of them just to move it along. This indicates a lack of interest to me. The
>>> problem is that we cannot really use a strict cut off date as a way to
>>> ensure people do the important things and skip the unimportant things,
>>> because we do know that the issues are critical.
>>> >
>>> > And, yes, the fact that Jenkins jobs are separately evolving but
>>> pretty tightly coupled to the repo contents is a serious problem that I
>>> wish we had fixed. So verification of each PR was manual.
>>> >
>>> > Altogether, I still think LTS is valuable to have as a promise to
>>> users that we will backport critical fixes. I would like to keep that
>>> promise and continue to try. Things that are rapidly changing (which
>>> something always will be) just won't have fixes backported, and that seems
>>> OK.
>>> >
>>> > Kenn
>>> >
>>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
>>> wrote:
>>> >>
>>> >> An LTS only makes sense if we end up patching the LTS, which so far we
>>> >> have never done. There has been work done in backporting fixes, see
>>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort
>>> was
>>> >> never completed. The main reason I believe were complications with
>>> >> running the evolved release scripts against old Beam versions.
>>> >>
>>> >> Now that the portability layer keeps maturing, it makes me optimistic
>>> >> that we might have a maintained LTS in the future.
>>> >>
>>> >> -Max
>>> >>
>>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>>> >> > The fact that end users never asked AFAIK in the ML for an LTS and
>>> for
>>> >> > a subsequent minor release of the existing LTS shows IMO the low
>>> >> > interest on having a LTS.
>>> >> >
>>> >> > We still are heavily iterating in many areas (portability/schema)
>>> and
>>> >> > I am not sure users (and in particular users of open source runners)
>>> >> > get a big benefit of relying on an old version. Maybe this is the
>>> >> > moment to reconsider if having a LTS does even make sense given (1)
>>> >> > that our end user facing APIs are 'mostly' stable (even if many
>>> still
>>> >> > called @Experimental). (2) that users get mostly improvements on
>>> >> > runners translation and newer APIs with a low cost just by updating
>>> >> > the version number, and (3) that in case of any regression in an
>>> >> > intermediary release we still can do a minor release even if we have
>>> >> > not yet done so, let's not forget that the only thing we need to do
>>> >> > this is enough interest to do the release from the maintainers.
>>> >> >
>>> >> >
>>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>>> >> > <va...@google.com> wrote:
>>> >> >>
>>> >> >> I support nominating 2.16.0 as LTS release since in has robust
>>> Python 3 support compared with prior releases, and also for reasons of
>>> pending Python 2 deprecation. This has been discussed before [1]. As Robert
>>> pointed out in that thread, LTS nomination in Beam is currently
>>> retroactive. If we keep the retroactive policy, the question is how long we
>>> should wait for a release to be considered "safe" for nomination.  Looks
>>> like in case of 2.7.0 we waited a month, see [2,3].
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Valentyn
>>> >> >>
>>> >> >> [1]
>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>>> >> >> [3]
>>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>> >> >>
>>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>>> whatwouldaustindo@gmail.com> wrote:
>>> >> >>>
>>> >> >>> Hi All,
>>> >> >>>
>>> >> >>> According to our policies page [1]: "There will be at least one
>>> new LTS release in a 12 month period, and LTS releases are considered
>>> deprecated after 12 months"
>>> >> >>>
>>> >> >>> The last LTS was released 2018-10-02 [2].
>>> >> >>>
>>> >> >>> Does that mean the next release (2.16) should be the next LTS?
>>> It looks like we are in danger of not living up to that promise.
>>> >> >>>
>>> >> >>> Cheers,
>>> >> >>> Austin
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> [1] https://beam.apache.org/community/policies/
>>> >> >>>
>>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>>
>>

Re: Next LTS?

[Austin Bennett <wh...@gmail.com>]

To be clear, I was picking on - or reminding us of - the promise: I don't
have a strong personal need/desire (at least currently) for LTS to exist.
Though, worth ensuring we live up to what we keep on the website.  And,
without an active LTS, probably something we should take off the site?

On Thu, Sep 19, 2019 at 1:33 PM Pablo Estrada <pa...@google.com> wrote:

> +Łukasz Gajowy <lu...@polidea.com> had at some point thought of
> setting up jenkins jobs without coupling them to the state of the repo
> during the last Seed Job. It may be that that improvement can help test
> older LTS-type releases?
>
> On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> In many ways the 2.7 LTS was trying to flesh out the process. I think
>> we learned some valuable lessons. It would have been good to push out
>> something (even if it didn't have everything we wanted) but that is
>> unlikely to be worth pursuing now (and 2.7 should probably be retired
>> as LTS and no longer recommended).
>>
>> I agree that it does not seem there is strong demand for an LTS at
>> this point. I would propose that we keep 2.16, etc. as potential
>> candidates, but only declare one as LTS pending demand. The question
>> of how to keep our tooling stable (or backwards/forwards compatible)
>> is a good one, especially as we move to drop Python 2.7 in 2020 (which
>> could itself be a driver for an LTS).
>>
>> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
>> >
>> > Yes, I pretty much dropped 2.7.1 release process due to lack of
>> interest.
>> >
>> > There are known problems so that I cannot recommend anyone to use
>> 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was
>> philosophical. I did not like the fact that we had a designated LTS family
>> with no usable releases.
>> >
>> > But many backports were proposed to block 2.7.1 and took a very long
>> time to get contirbutors to implement the backports. I ended up doing many
>> of them just to move it along. This indicates a lack of interest to me. The
>> problem is that we cannot really use a strict cut off date as a way to
>> ensure people do the important things and skip the unimportant things,
>> because we do know that the issues are critical.
>> >
>> > And, yes, the fact that Jenkins jobs are separately evolving but pretty
>> tightly coupled to the repo contents is a serious problem that I wish we
>> had fixed. So verification of each PR was manual.
>> >
>> > Altogether, I still think LTS is valuable to have as a promise to users
>> that we will backport critical fixes. I would like to keep that promise and
>> continue to try. Things that are rapidly changing (which something always
>> will be) just won't have fixes backported, and that seems OK.
>> >
>> > Kenn
>> >
>> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>
>> >> An LTS only makes sense if we end up patching the LTS, which so far we
>> >> have never done. There has been work done in backporting fixes, see
>> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort
>> was
>> >> never completed. The main reason I believe were complications with
>> >> running the evolved release scripts against old Beam versions.
>> >>
>> >> Now that the portability layer keeps maturing, it makes me optimistic
>> >> that we might have a maintained LTS in the future.
>> >>
>> >> -Max
>> >>
>> >> On 19.09.19 08:40, Ismaël Mejía wrote:
>> >> > The fact that end users never asked AFAIK in the ML for an LTS and
>> for
>> >> > a subsequent minor release of the existing LTS shows IMO the low
>> >> > interest on having a LTS.
>> >> >
>> >> > We still are heavily iterating in many areas (portability/schema) and
>> >> > I am not sure users (and in particular users of open source runners)
>> >> > get a big benefit of relying on an old version. Maybe this is the
>> >> > moment to reconsider if having a LTS does even make sense given (1)
>> >> > that our end user facing APIs are 'mostly' stable (even if many still
>> >> > called @Experimental). (2) that users get mostly improvements on
>> >> > runners translation and newer APIs with a low cost just by updating
>> >> > the version number, and (3) that in case of any regression in an
>> >> > intermediary release we still can do a minor release even if we have
>> >> > not yet done so, let's not forget that the only thing we need to do
>> >> > this is enough interest to do the release from the maintainers.
>> >> >
>> >> >
>> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>> >> > <va...@google.com> wrote:
>> >> >>
>> >> >> I support nominating 2.16.0 as LTS release since in has robust
>> Python 3 support compared with prior releases, and also for reasons of
>> pending Python 2 deprecation. This has been discussed before [1]. As Robert
>> pointed out in that thread, LTS nomination in Beam is currently
>> retroactive. If we keep the retroactive policy, the question is how long we
>> should wait for a release to be considered "safe" for nomination.  Looks
>> like in case of 2.7.0 we waited a month, see [2,3].
>> >> >>
>> >> >> Thanks,
>> >> >> Valentyn
>> >> >>
>> >> >> [1]
>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>> >> >> [3]
>> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>> >> >>
>> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
>> whatwouldaustindo@gmail.com> wrote:
>> >> >>>
>> >> >>> Hi All,
>> >> >>>
>> >> >>> According to our policies page [1]: "There will be at least one
>> new LTS release in a 12 month period, and LTS releases are considered
>> deprecated after 12 months"
>> >> >>>
>> >> >>> The last LTS was released 2018-10-02 [2].
>> >> >>>
>> >> >>> Does that mean the next release (2.16) should be the next LTS?  It
>> looks like we are in danger of not living up to that promise.
>> >> >>>
>> >> >>> Cheers,
>> >> >>> Austin
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> [1] https://beam.apache.org/community/policies/
>> >> >>>
>> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>>
>

Re: Next LTS?

[Pablo Estrada <pa...@google.com>]

+Łukasz Gajowy <lu...@polidea.com> had at some point thought of
setting up jenkins jobs without coupling them to the state of the repo
during the last Seed Job. It may be that that improvement can help test
older LTS-type releases?

On Thu, Sep 19, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com> wrote:

> In many ways the 2.7 LTS was trying to flesh out the process. I think
> we learned some valuable lessons. It would have been good to push out
> something (even if it didn't have everything we wanted) but that is
> unlikely to be worth pursuing now (and 2.7 should probably be retired
> as LTS and no longer recommended).
>
> I agree that it does not seem there is strong demand for an LTS at
> this point. I would propose that we keep 2.16, etc. as potential
> candidates, but only declare one as LTS pending demand. The question
> of how to keep our tooling stable (or backwards/forwards compatible)
> is a good one, especially as we move to drop Python 2.7 in 2020 (which
> could itself be a driver for an LTS).
>
> On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
> >
> > Yes, I pretty much dropped 2.7.1 release process due to lack of interest.
> >
> > There are known problems so that I cannot recommend anyone to use 2.7.0,
> yet 2.7 it is the current LTS family. So my work on 2.7.1 was
> philosophical. I did not like the fact that we had a designated LTS family
> with no usable releases.
> >
> > But many backports were proposed to block 2.7.1 and took a very long
> time to get contirbutors to implement the backports. I ended up doing many
> of them just to move it along. This indicates a lack of interest to me. The
> problem is that we cannot really use a strict cut off date as a way to
> ensure people do the important things and skip the unimportant things,
> because we do know that the issues are critical.
> >
> > And, yes, the fact that Jenkins jobs are separately evolving but pretty
> tightly coupled to the repo contents is a serious problem that I wish we
> had fixed. So verification of each PR was manual.
> >
> > Altogether, I still think LTS is valuable to have as a promise to users
> that we will backport critical fixes. I would like to keep that promise and
> continue to try. Things that are rapidly changing (which something always
> will be) just won't have fixes backported, and that seems OK.
> >
> > Kenn
> >
> > On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> An LTS only makes sense if we end up patching the LTS, which so far we
> >> have never done. There has been work done in backporting fixes, see
> >> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
> >> never completed. The main reason I believe were complications with
> >> running the evolved release scripts against old Beam versions.
> >>
> >> Now that the portability layer keeps maturing, it makes me optimistic
> >> that we might have a maintained LTS in the future.
> >>
> >> -Max
> >>
> >> On 19.09.19 08:40, Ismaël Mejía wrote:
> >> > The fact that end users never asked AFAIK in the ML for an LTS and for
> >> > a subsequent minor release of the existing LTS shows IMO the low
> >> > interest on having a LTS.
> >> >
> >> > We still are heavily iterating in many areas (portability/schema) and
> >> > I am not sure users (and in particular users of open source runners)
> >> > get a big benefit of relying on an old version. Maybe this is the
> >> > moment to reconsider if having a LTS does even make sense given (1)
> >> > that our end user facing APIs are 'mostly' stable (even if many still
> >> > called @Experimental). (2) that users get mostly improvements on
> >> > runners translation and newer APIs with a low cost just by updating
> >> > the version number, and (3) that in case of any regression in an
> >> > intermediary release we still can do a minor release even if we have
> >> > not yet done so, let's not forget that the only thing we need to do
> >> > this is enough interest to do the release from the maintainers.
> >> >
> >> >
> >> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> >> > <va...@google.com> wrote:
> >> >>
> >> >> I support nominating 2.16.0 as LTS release since in has robust
> Python 3 support compared with prior releases, and also for reasons of
> pending Python 2 deprecation. This has been discussed before [1]. As Robert
> pointed out in that thread, LTS nomination in Beam is currently
> retroactive. If we keep the retroactive policy, the question is how long we
> should wait for a release to be considered "safe" for nomination.  Looks
> like in case of 2.7.0 we waited a month, see [2,3].
> >> >>
> >> >> Thanks,
> >> >> Valentyn
> >> >>
> >> >> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> >> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> >> >> [3]
> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
> >> >>
> >> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> >> >>>
> >> >>> Hi All,
> >> >>>
> >> >>> According to our policies page [1]: "There will be at least one new
> LTS release in a 12 month period, and LTS releases are considered
> deprecated after 12 months"
> >> >>>
> >> >>> The last LTS was released 2018-10-02 [2].
> >> >>>
> >> >>> Does that mean the next release (2.16) should be the next LTS?  It
> looks like we are in danger of not living up to that promise.
> >> >>>
> >> >>> Cheers,
> >> >>> Austin
> >> >>>
> >> >>>
> >> >>>
> >> >>> [1] https://beam.apache.org/community/policies/
> >> >>>
> >> >>> [2]  https://beam.apache.org/get-started/downloads/
>

Re: Next LTS?

[Robert Bradshaw <ro...@google.com>]

In many ways the 2.7 LTS was trying to flesh out the process. I think
we learned some valuable lessons. It would have been good to push out
something (even if it didn't have everything we wanted) but that is
unlikely to be worth pursuing now (and 2.7 should probably be retired
as LTS and no longer recommended).

I agree that it does not seem there is strong demand for an LTS at
this point. I would propose that we keep 2.16, etc. as potential
candidates, but only declare one as LTS pending demand. The question
of how to keep our tooling stable (or backwards/forwards compatible)
is a good one, especially as we move to drop Python 2.7 in 2020 (which
could itself be a driver for an LTS).

On Thu, Sep 19, 2019 at 12:27 PM Kenneth Knowles <ke...@apache.org> wrote:
>
> Yes, I pretty much dropped 2.7.1 release process due to lack of interest.
>
> There are known problems so that I cannot recommend anyone to use 2.7.0, yet 2.7 it is the current LTS family. So my work on 2.7.1 was philosophical. I did not like the fact that we had a designated LTS family with no usable releases.
>
> But many backports were proposed to block 2.7.1 and took a very long time to get contirbutors to implement the backports. I ended up doing many of them just to move it along. This indicates a lack of interest to me. The problem is that we cannot really use a strict cut off date as a way to ensure people do the important things and skip the unimportant things, because we do know that the issues are critical.
>
> And, yes, the fact that Jenkins jobs are separately evolving but pretty tightly coupled to the repo contents is a serious problem that I wish we had fixed. So verification of each PR was manual.
>
> Altogether, I still think LTS is valuable to have as a promise to users that we will backport critical fixes. I would like to keep that promise and continue to try. Things that are rapidly changing (which something always will be) just won't have fixes backported, and that seems OK.
>
> Kenn
>
> On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org> wrote:
>>
>> An LTS only makes sense if we end up patching the LTS, which so far we
>> have never done. There has been work done in backporting fixes, see
>> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
>> never completed. The main reason I believe were complications with
>> running the evolved release scripts against old Beam versions.
>>
>> Now that the portability layer keeps maturing, it makes me optimistic
>> that we might have a maintained LTS in the future.
>>
>> -Max
>>
>> On 19.09.19 08:40, Ismaël Mejía wrote:
>> > The fact that end users never asked AFAIK in the ML for an LTS and for
>> > a subsequent minor release of the existing LTS shows IMO the low
>> > interest on having a LTS.
>> >
>> > We still are heavily iterating in many areas (portability/schema) and
>> > I am not sure users (and in particular users of open source runners)
>> > get a big benefit of relying on an old version. Maybe this is the
>> > moment to reconsider if having a LTS does even make sense given (1)
>> > that our end user facing APIs are 'mostly' stable (even if many still
>> > called @Experimental). (2) that users get mostly improvements on
>> > runners translation and newer APIs with a low cost just by updating
>> > the version number, and (3) that in case of any regression in an
>> > intermediary release we still can do a minor release even if we have
>> > not yet done so, let's not forget that the only thing we need to do
>> > this is enough interest to do the release from the maintainers.
>> >
>> >
>> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
>> > <va...@google.com> wrote:
>> >>
>> >> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
>> >>
>> >> Thanks,
>> >> Valentyn
>> >>
>> >> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>> >> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>> >>
>> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
>> >>>
>> >>> Hi All,
>> >>>
>> >>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
>> >>>
>> >>> The last LTS was released 2018-10-02 [2].
>> >>>
>> >>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
>> >>>
>> >>> Cheers,
>> >>> Austin
>> >>>
>> >>>
>> >>>
>> >>> [1] https://beam.apache.org/community/policies/
>> >>>
>> >>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Kenneth Knowles <ke...@apache.org>]

Yes, I pretty much dropped 2.7.1 release process due to lack of interest.

There are known problems so that I cannot recommend anyone to use 2.7.0,
yet 2.7 it is the current LTS family. So my work on 2.7.1 was
philosophical. I did not like the fact that we had a designated LTS family
with no usable releases.

But many backports were proposed to block 2.7.1 and took a very long time
to get contirbutors to implement the backports. I ended up doing many of
them just to move it along. This indicates a lack of interest to me. The
problem is that we cannot really use a strict cut off date as a way to
ensure people do the important things and skip the unimportant things,
because we do know that the issues are critical.

And, yes, the fact that Jenkins jobs are separately evolving but pretty
tightly coupled to the repo contents is a serious problem that I wish we
had fixed. So verification of each PR was manual.

Altogether, I still think LTS is valuable to have as a promise to users
that we will backport critical fixes. I would like to keep that promise and
continue to try. Things that are rapidly changing (which something always
will be) just won't have fixes backported, and that seems OK.

Kenn

On Thu, Sep 19, 2019 at 10:59 AM Maximilian Michels <mx...@apache.org> wrote:

> An LTS only makes sense if we end up patching the LTS, which so far we
> have never done. There has been work done in backporting fixes, see
> https://github.com/apache/beam/commits/release-2.7.1 but the effort was
> never completed. The main reason I believe were complications with
> running the evolved release scripts against old Beam versions.
>
> Now that the portability layer keeps maturing, it makes me optimistic
> that we might have a maintained LTS in the future.
>
> -Max
>
> On 19.09.19 08:40, Ismaël Mejía wrote:
> > The fact that end users never asked AFAIK in the ML for an LTS and for
> > a subsequent minor release of the existing LTS shows IMO the low
> > interest on having a LTS.
> >
> > We still are heavily iterating in many areas (portability/schema) and
> > I am not sure users (and in particular users of open source runners)
> > get a big benefit of relying on an old version. Maybe this is the
> > moment to reconsider if having a LTS does even make sense given (1)
> > that our end user facing APIs are 'mostly' stable (even if many still
> > called @Experimental). (2) that users get mostly improvements on
> > runners translation and newer APIs with a low cost just by updating
> > the version number, and (3) that in case of any regression in an
> > intermediary release we still can do a minor release even if we have
> > not yet done so, let's not forget that the only thing we need to do
> > this is enough interest to do the release from the maintainers.
> >
> >
> > On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> > <va...@google.com> wrote:
> >>
> >> I support nominating 2.16.0 as LTS release since in has robust Python 3
> support compared with prior releases, and also for reasons of pending
> Python 2 deprecation. This has been discussed before [1]. As Robert pointed
> out in that thread, LTS nomination in Beam is currently retroactive. If we
> keep the retroactive policy, the question is how long we should wait for a
> release to be considered "safe" for nomination.  Looks like in case of
> 2.7.0 we waited a month, see [2,3].
> >>
> >> Thanks,
> >> Valentyn
> >>
> >> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> >> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> >> [3]
> https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
> >>
> >> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <
> whatwouldaustindo@gmail.com> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> According to our policies page [1]: "There will be at least one new
> LTS release in a 12 month period, and LTS releases are considered
> deprecated after 12 months"
> >>>
> >>> The last LTS was released 2018-10-02 [2].
> >>>
> >>> Does that mean the next release (2.16) should be the next LTS?  It
> looks like we are in danger of not living up to that promise.
> >>>
> >>> Cheers,
> >>> Austin
> >>>
> >>>
> >>>
> >>> [1] https://beam.apache.org/community/policies/
> >>>
> >>> [2]  https://beam.apache.org/get-started/downloads/
>

Re: Next LTS?

[Maximilian Michels <mx...@apache.org>]

An LTS only makes sense if we end up patching the LTS, which so far we 
have never done. There has been work done in backporting fixes, see 
https://github.com/apache/beam/commits/release-2.7.1 but the effort was 
never completed. The main reason I believe were complications with 
running the evolved release scripts against old Beam versions.

Now that the portability layer keeps maturing, it makes me optimistic 
that we might have a maintained LTS in the future.

-Max

On 19.09.19 08:40, Ismaël Mejía wrote:
> The fact that end users never asked AFAIK in the ML for an LTS and for
> a subsequent minor release of the existing LTS shows IMO the low
> interest on having a LTS.
> 
> We still are heavily iterating in many areas (portability/schema) and
> I am not sure users (and in particular users of open source runners)
> get a big benefit of relying on an old version. Maybe this is the
> moment to reconsider if having a LTS does even make sense given (1)
> that our end user facing APIs are 'mostly' stable (even if many still
> called @Experimental). (2) that users get mostly improvements on
> runners translation and newer APIs with a low cost just by updating
> the version number, and (3) that in case of any regression in an
> intermediary release we still can do a minor release even if we have
> not yet done so, let's not forget that the only thing we need to do
> this is enough interest to do the release from the maintainers.
> 
> 
> On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
> <va...@google.com> wrote:
>>
>> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
>>
>> Thanks,
>> Valentyn
>>
>> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
>> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>>
>> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
>>>
>>> The last LTS was released 2018-10-02 [2].
>>>
>>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
>>>
>>> Cheers,
>>> Austin
>>>
>>>
>>>
>>> [1] https://beam.apache.org/community/policies/
>>>
>>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Ismaël Mejía <ie...@gmail.com>]

The fact that end users never asked AFAIK in the ML for an LTS and for
a subsequent minor release of the existing LTS shows IMO the low
interest on having a LTS.

We still are heavily iterating in many areas (portability/schema) and
I am not sure users (and in particular users of open source runners)
get a big benefit of relying on an old version. Maybe this is the
moment to reconsider if having a LTS does even make sense given (1)
that our end user facing APIs are 'mostly' stable (even if many still
called @Experimental). (2) that users get mostly improvements on
runners translation and newer APIs with a low cost just by updating
the version number, and (3) that in case of any regression in an
intermediary release we still can do a minor release even if we have
not yet done so, let's not forget that the only thing we need to do
this is enough interest to do the release from the maintainers.


On Tue, Sep 17, 2019 at 12:00 AM Valentyn Tymofieiev
<va...@google.com> wrote:
>
> I support nominating 2.16.0 as LTS release since in has robust Python 3 support compared with prior releases, and also for reasons of pending Python 2 deprecation. This has been discussed before [1]. As Robert pointed out in that thread, LTS nomination in Beam is currently retroactive. If we keep the retroactive policy, the question is how long we should wait for a release to be considered "safe" for nomination.  Looks like in case of 2.7.0 we waited a month, see [2,3].
>
> Thanks,
> Valentyn
>
> [1] https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> [2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
> [3] https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E
>
> On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com> wrote:
>>
>> Hi All,
>>
>> According to our policies page [1]: "There will be at least one new LTS release in a 12 month period, and LTS releases are considered deprecated after 12 months"
>>
>> The last LTS was released 2018-10-02 [2].
>>
>> Does that mean the next release (2.16) should be the next LTS?  It looks like we are in danger of not living up to that promise.
>>
>> Cheers,
>> Austin
>>
>>
>>
>> [1] https://beam.apache.org/community/policies/
>>
>> [2]  https://beam.apache.org/get-started/downloads/

Re: Next LTS?

[Valentyn Tymofieiev <va...@google.com>]

I support nominating 2.16.0 as LTS release since in has robust Python 3
support compared with prior releases, and also for reasons of pending
Python 2 deprecation. This has been discussed before [1]. As Robert pointed
out in that thread, LTS nomination in Beam is currently retroactive. If we
keep the retroactive policy, the question is how long we should wait for a
release to be considered "safe" for nomination.  Looks like in case of
2.7.0 we waited a month, see [2,3].

Thanks,
Valentyn

[1]
https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
[2] https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html
[3]
https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E

On Mon, Sep 16, 2019 at 2:46 PM Austin Bennett <wh...@gmail.com>
wrote:

> Hi All,
>
> According to our policies page [1]: "There will be at least one new LTS
> release in a 12 month period, and LTS releases are considered deprecated
> after 12 months"
>
> The last LTS was released 2018-10-02 [2].
>
> Does that mean the next release (2.16) should be the next LTS?  It looks
> like we are in danger of not living up to that promise.
>
> Cheers,
> Austin
>
>
>
> [1] https://beam.apache.org/community/policies/
>
> [2]  https://beam.apache.org/get-started/downloads/
>