[DISCUSS] EOL of BookKeeper releases

[Sijie Guo <gu...@gmail.com>]

Hi all,

We have switched to Time Based Release Schedule
<https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-13+-+Time+Based+Release+Plan>
since
4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
4.6.1. And the 4.7.0 release is coming in a month or so. According to
BP-13, we need to enforce an EOL policy for our releases: keep the releases
in one year and mark the releases older than one year as EOL, no BC would
be considered with those versions.

I would like to propose we start enforcing EOL at 4.7.0 release:

- we only keep all the releases higher than 4.4.0 (4.4.0 was released about
one year and half ago).
- stop supporting BC with releases smaller than 4.4.0. This includes: move
the documentation of those releases as archived, remove them from BC tests,
delete the legacy code that was there for supporting BC with these releases.

Thoughts?

- Sijie

Re: [DISCUSS] EOL of BookKeeper releases

[Enrico Olivelli <eo...@gmail.com>]

Il gio 1 feb 2018, 08:17 Sijie Guo <gu...@gmail.com> ha scritto:

> Hi all,
>
> We have switched to Time Based Release Schedule
> <
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-13+-+Time+Based+Release+Plan
> >
> since
> 4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
> 4.6.1. And the 4.7.0 release is coming in a month or so. According to
> BP-13, we need to enforce an EOL policy for our releases: keep the releases
> in one year and mark the releases older than one year as EOL, no BC would
> be considered with those versions.
>
> I would like to propose we start enforcing EOL at 4.7.0 release:
>
> - we only keep all the releases higher than 4.4.0 (4.4.0 was released about
> one year and half ago).
> - stop supporting BC with releases smaller than 4.4.0. This includes: move
> the documentation of those releases as archived, remove them from BC tests,
> delete the legacy code that was there for supporting BC with these
> releases.
>
> Thoughts?
>

Very good
+1

Enrico


> - Sijie
>


-- 


-- Enrico Olivelli

Re: [DISCUSS] EOL of BookKeeper releases

[Sijie Guo <gu...@gmail.com>]

On Thu, Feb 1, 2018 at 4:34 AM, Enrico Olivelli <eo...@gmail.com> wrote:

> 2018-02-01 13:29 GMT+01:00 Sijie Guo <gu...@gmail.com>:
>
> > On Thu, Feb 1, 2018 at 2:51 AM, Ivan Kelly <iv...@apache.org> wrote:
> >
> > > > We have switched to Time Based Release Schedule
> > > > <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> > > BP-13+-+Time+Based+Release+Plan>
> > > > since
> > > > 4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
> > > > 4.6.1. And the 4.7.0 release is coming in a month or so. According to
> > > > BP-13, we need to enforce an EOL policy for our releases: keep the
> > > releases
> > > > in one year and mark the releases older than one year as EOL, no BC
> > would
> > > > be considered with those versions.
> > > Perhaps we should define an EOL policy before EOLing releases. As part
> > > of this, we should define what we support for BC and what we don't.
> > > For example, do we support BC on all ledger managers? Just
> > > hierarchical and flat?
> > >
> > > > I would like to propose we start enforcing EOL at 4.7.0 release:
> > > +1
> > >
> > > > - we only keep all the releases higher than 4.4.0 (4.4.0 was released
> > > about
> > > > one year and half ago).
> > > inclusive or exclusive of 4.4.0?
> > >
> >
> > Inclusive.
> >
> > >
> > > > - stop supporting BC with releases smaller than 4.4.0. This includes:
> > > move
> > > > the documentation of those releases as archived, remove them from BC
> > > tests,
> > > > delete the legacy code that was there for supporting BC with these
> > > releases.
> > > What do we mean by stopping support for BC?
> >
> >
> > It means we will NOT consider BC problems with those old releases.
> >
> >
> > > We haven't broken client
> > > compatibility since 4.1.0. I don't think we have any special code for
> > > handling it. But I think there is special code for handling old ledger
> > > metadata formats (which haven't been upgrades). If someone started
> > > their cluster on version 4.1.0, and upgraded version by version to
> > > 4.6, then the ledger metadata from 4.1.0 will still be there, and
> > > their 4.6.0 client will be depending on the legacy code paths to
> > > function.
> >
> > I don't think it's as simple as just removing the legacy
> > > code, and it's something we need to be very careful about.
> >
> >
> > It is case by case.
> >
> > for example, the password and digest type was introduced at a very old
> > release.
> > There is code assuming ledgers might not have password or digest.
> However,
> > the releases onwards would already have those fields.
> > If we stop supporting those old version, we can remove that special BC
> > code, which will make code clearer to maintain.
> >
> >
> > > The BC
> > > tests should help in this regard, but we need keep testing scenarios
> > > where the original cluster was brought up in the oldest version
> > > (4.1.0).
> > >
> >
> > If we stop supporting 4.1.0, we don't need to test those scenarios. We
> > started with from the lowest version we are claiming to support.
> > That is the whole point of having EOL.
> >
>
>
> I agree with Ivan, there can be old clusters with legacy metadata.
>
> We should provide some tool to upgrade ZK (meta) data to the latest
> version.
>

Yes we need this tool when we enforce EOL. I didn't say we don't need this
tool.


>
> I think that local Bookie data are not a issue as bookie usually
> automatically upgrades data on first boot with new version (I can be wrong,
> I am following BK only since 4.3)
>
> Enrico
>
>
>
>
> >
> >
> > >
> > > -Ivan
> > >
> >
>

Re: [DISCUSS] EOL of BookKeeper releases

[Enrico Olivelli <eo...@gmail.com>]

2018-02-01 13:29 GMT+01:00 Sijie Guo <gu...@gmail.com>:

> On Thu, Feb 1, 2018 at 2:51 AM, Ivan Kelly <iv...@apache.org> wrote:
>
> > > We have switched to Time Based Release Schedule
> > > <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> > BP-13+-+Time+Based+Release+Plan>
> > > since
> > > 4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
> > > 4.6.1. And the 4.7.0 release is coming in a month or so. According to
> > > BP-13, we need to enforce an EOL policy for our releases: keep the
> > releases
> > > in one year and mark the releases older than one year as EOL, no BC
> would
> > > be considered with those versions.
> > Perhaps we should define an EOL policy before EOLing releases. As part
> > of this, we should define what we support for BC and what we don't.
> > For example, do we support BC on all ledger managers? Just
> > hierarchical and flat?
> >
> > > I would like to propose we start enforcing EOL at 4.7.0 release:
> > +1
> >
> > > - we only keep all the releases higher than 4.4.0 (4.4.0 was released
> > about
> > > one year and half ago).
> > inclusive or exclusive of 4.4.0?
> >
>
> Inclusive.
>
> >
> > > - stop supporting BC with releases smaller than 4.4.0. This includes:
> > move
> > > the documentation of those releases as archived, remove them from BC
> > tests,
> > > delete the legacy code that was there for supporting BC with these
> > releases.
> > What do we mean by stopping support for BC?
>
>
> It means we will NOT consider BC problems with those old releases.
>
>
> > We haven't broken client
> > compatibility since 4.1.0. I don't think we have any special code for
> > handling it. But I think there is special code for handling old ledger
> > metadata formats (which haven't been upgrades). If someone started
> > their cluster on version 4.1.0, and upgraded version by version to
> > 4.6, then the ledger metadata from 4.1.0 will still be there, and
> > their 4.6.0 client will be depending on the legacy code paths to
> > function.
>
> I don't think it's as simple as just removing the legacy
> > code, and it's something we need to be very careful about.
>
>
> It is case by case.
>
> for example, the password and digest type was introduced at a very old
> release.
> There is code assuming ledgers might not have password or digest. However,
> the releases onwards would already have those fields.
> If we stop supporting those old version, we can remove that special BC
> code, which will make code clearer to maintain.
>
>
> > The BC
> > tests should help in this regard, but we need keep testing scenarios
> > where the original cluster was brought up in the oldest version
> > (4.1.0).
> >
>
> If we stop supporting 4.1.0, we don't need to test those scenarios. We
> started with from the lowest version we are claiming to support.
> That is the whole point of having EOL.
>


I agree with Ivan, there can be old clusters with legacy metadata.

We should provide some tool to upgrade ZK (meta) data to the latest version.

I think that local Bookie data are not a issue as bookie usually
automatically upgrades data on first boot with new version (I can be wrong,
I am following BK only since 4.3)

Enrico




>
>
> >
> > -Ivan
> >
>

Re: [DISCUSS] EOL of BookKeeper releases

[Sijie Guo <gu...@gmail.com>]

On Thu, Feb 1, 2018 at 12:03 PM, Ivan Kelly <iv...@apache.org> wrote:

> Ok, I think we're on the same page as to what is needed. The tool
> needs to be in place before any version is EOL.
>
> I do think we should retain some BC tests with the older versions, of the
> form:
> - init cluster with 4.1.0

- run 4.7.0 upgrade tool
> - start latest (4.8.0-SNAPSHOT) bookie and client
>

that is fine.


>
> Another thing to consider is that ledger metadata upgrade will involve
> downtime for the client. This is fine, they can have a maintenance
> window, but we should be clear in the documentation that this is
> needed.
>
> -Ivan
>
> On Thu, Feb 1, 2018 at 5:58 PM, Sijie Guo <gu...@gmail.com> wrote:
> > I think you misunderstood what I was saying.
> >
> >
> > On Thu, Feb 1, 2018 at 4:53 AM, Ivan Kelly <iv...@apache.org> wrote:
> >
> >> >> What do we mean by stopping support for BC?
> >> >
> >> > It means we will NOT consider BC problems with those old releases.
> >>
> >> If EOL means ignoring all BC considerations, then I'm -1 on these
> >> proposals. But I don't think we need to go to those extremes.
> >> I think we need to specify what BC guarantees we give on any version,
> >> and then EOL policies will follow from that. The 3 big BC
> >> considerations, as I see it, are:
> >> 1) Data on the bookie
> >> 2) Client wire protocol
> >> 3) Data shared between bookies and clients (i.e. zookeeper metadata)
> >>
> >> For 1 we can have quite a short timeframe for EOL old software.
> >> Bookies can be upgraded individually, and this is controlled by the
> >> administrator.
> >> For 2 we need a bit longer, because within an organisation there may
> >> be many different versions of the client knocking around and it's hard
> >> to get everyone to upgrade.
> >> For 3 we can pretty much never have a hard break. If a cluster was
> >> initialized with an old version, they must be able to able to upgrade
> >> that cluster to the latest version of the software. Otherwise they
> >> will stick with the old version forever.
> >>
> >
> > when I am saying stop all BC consideration, I mean:
> >
> > 1) all the data and metadata will be upgraded to the minimal version we
> > claim to support.
> > 2) all the client would have to upgrade to the minimal version we claim
> to
> > support for wire protocol.
> >
> > Let's take 4.7.0 as the example, when I am saying enforce EOL at 4.7.0.
> It
> > means 4.7.0 will still compatible with 4.1.0. but at 4.7.0 we are
> thinking
> > about dropping 4.1.x, 4.2.x and 4.3.x.
> >
> > so at 4.7.0:
> >
> > - we will have to develop a tool to upgrade metadata and data to the
> > version that will be recognized by 4.4.x and onwards releases.
> >
> > so at 4.8.0:
> >
> > - we will not consider clients older than 4.4.x. all the legal code can
> be
> > potentially removed.
> >
> > That means if a cluster (either metadata/data produced by clients older
> > than 4.4.x or still have old clients) want to upgrade to 4.8.0, it has to
> > first upgrade the cluster (both metadata and data) to 4.7.0 following the
> > instructions provided at 4.7.0
> > and upgrade all the clients to minimal version that 4.8.0 will support.
> > After all is done first, the cluster can be upgraded to 4.8.0.
> >
> >
> > Hope this example make things clear.
> >
> > Enforcing EOL means dropping the BC considerations between the oldest
> > version and newest version and limit the BC considerations within a
> limited
> > number of versions which is more controllable, maintenanable. Using the
> > example above,
> > it means if you want to upgrade from 4.7.x to 4.8.x, please use the tools
> > provided in 4.7.x to ensure metadata/data are upgraded to latest version
> > and no more clients older than 4.4.0 talk to the cluster.
> >
> >
> >>
> >> > for example, the password and digest type was introduced at a very old
> >> > release.
> >> > There is code assuming ledgers might not have password or digest.
> >> However,
> >> > the releases onwards would already have those fields.
> >> > If we stop supporting those old version, we can remove that special BC
> >> > code, which will make code clearer to maintain.
> >>
> >> This code is in the client. I'm fine with stopping support for all
> >> clients older than a certain era. However, yahoo is still on 4.3, so
> >> we should at least support back that far until they've been migrated
> >> forward.
> >>
> >
> > the yahoo branch is between 4.3 and 4.4. it is a special case that we
> need
> > to take care of. we are not dropping it out of consideration radar.
> >
> >
> >>
> >> >> The BC
> >> >> tests should help in this regard, but we need keep testing scenarios
> >> >> where the original cluster was brought up in the oldest version
> >> >> (4.1.0).
> >> >
> >> > If we stop supporting 4.1.0, we don't need to test those scenarios. We
> >> > started with from the lowest version we are claiming to support.
> >>
> >> Are we supporting the software or data that was created with that
> >> software? That's two different things. Sure, we're not going to fix
> >> bugs on pre-4.4.0 releases, but there are pre-4.4.0 clusters running
> >> in the wild (yahoo being an example), and they have to be usable with
> >> the latest software, otherwise they will never upgrade. I had this in
> >> my previous job where they made a big breaking BC change in the
> >> metadata between two versions. While it did allow us to delete a lot
> >> of code, we ended up supporting two codebases forever, as customers
> >> were unable to upgrade to the latest version.
> >>
> >
> > Please check my example above to see if that makes sense to you.
> >
> >
> >>
> >> > That is the whole point of having EOL.
> >> It's not clear what the point of having EOL is, since we don't have a
> >> clearly defined EOL policy nor BC policy.
> >>
> >
> > Please check my example above to see if that makes sense to you.
> >
> >
> >>
> >> -Ivan
> >>
>

Re: [DISCUSS] EOL of BookKeeper releases

[Ivan Kelly <iv...@apache.org>]

Ok, I think we're on the same page as to what is needed. The tool
needs to be in place before any version is EOL.

I do think we should retain some BC tests with the older versions, of the form:
- init cluster with 4.1.0
- run 4.7.0 upgrade tool
- start latest (4.8.0-SNAPSHOT) bookie and client

Another thing to consider is that ledger metadata upgrade will involve
downtime for the client. This is fine, they can have a maintenance
window, but we should be clear in the documentation that this is
needed.

-Ivan

On Thu, Feb 1, 2018 at 5:58 PM, Sijie Guo <gu...@gmail.com> wrote:
> I think you misunderstood what I was saying.
>
>
> On Thu, Feb 1, 2018 at 4:53 AM, Ivan Kelly <iv...@apache.org> wrote:
>
>> >> What do we mean by stopping support for BC?
>> >
>> > It means we will NOT consider BC problems with those old releases.
>>
>> If EOL means ignoring all BC considerations, then I'm -1 on these
>> proposals. But I don't think we need to go to those extremes.
>> I think we need to specify what BC guarantees we give on any version,
>> and then EOL policies will follow from that. The 3 big BC
>> considerations, as I see it, are:
>> 1) Data on the bookie
>> 2) Client wire protocol
>> 3) Data shared between bookies and clients (i.e. zookeeper metadata)
>>
>> For 1 we can have quite a short timeframe for EOL old software.
>> Bookies can be upgraded individually, and this is controlled by the
>> administrator.
>> For 2 we need a bit longer, because within an organisation there may
>> be many different versions of the client knocking around and it's hard
>> to get everyone to upgrade.
>> For 3 we can pretty much never have a hard break. If a cluster was
>> initialized with an old version, they must be able to able to upgrade
>> that cluster to the latest version of the software. Otherwise they
>> will stick with the old version forever.
>>
>
> when I am saying stop all BC consideration, I mean:
>
> 1) all the data and metadata will be upgraded to the minimal version we
> claim to support.
> 2) all the client would have to upgrade to the minimal version we claim to
> support for wire protocol.
>
> Let's take 4.7.0 as the example, when I am saying enforce EOL at 4.7.0. It
> means 4.7.0 will still compatible with 4.1.0. but at 4.7.0 we are thinking
> about dropping 4.1.x, 4.2.x and 4.3.x.
>
> so at 4.7.0:
>
> - we will have to develop a tool to upgrade metadata and data to the
> version that will be recognized by 4.4.x and onwards releases.
>
> so at 4.8.0:
>
> - we will not consider clients older than 4.4.x. all the legal code can be
> potentially removed.
>
> That means if a cluster (either metadata/data produced by clients older
> than 4.4.x or still have old clients) want to upgrade to 4.8.0, it has to
> first upgrade the cluster (both metadata and data) to 4.7.0 following the
> instructions provided at 4.7.0
> and upgrade all the clients to minimal version that 4.8.0 will support.
> After all is done first, the cluster can be upgraded to 4.8.0.
>
>
> Hope this example make things clear.
>
> Enforcing EOL means dropping the BC considerations between the oldest
> version and newest version and limit the BC considerations within a limited
> number of versions which is more controllable, maintenanable. Using the
> example above,
> it means if you want to upgrade from 4.7.x to 4.8.x, please use the tools
> provided in 4.7.x to ensure metadata/data are upgraded to latest version
> and no more clients older than 4.4.0 talk to the cluster.
>
>
>>
>> > for example, the password and digest type was introduced at a very old
>> > release.
>> > There is code assuming ledgers might not have password or digest.
>> However,
>> > the releases onwards would already have those fields.
>> > If we stop supporting those old version, we can remove that special BC
>> > code, which will make code clearer to maintain.
>>
>> This code is in the client. I'm fine with stopping support for all
>> clients older than a certain era. However, yahoo is still on 4.3, so
>> we should at least support back that far until they've been migrated
>> forward.
>>
>
> the yahoo branch is between 4.3 and 4.4. it is a special case that we need
> to take care of. we are not dropping it out of consideration radar.
>
>
>>
>> >> The BC
>> >> tests should help in this regard, but we need keep testing scenarios
>> >> where the original cluster was brought up in the oldest version
>> >> (4.1.0).
>> >
>> > If we stop supporting 4.1.0, we don't need to test those scenarios. We
>> > started with from the lowest version we are claiming to support.
>>
>> Are we supporting the software or data that was created with that
>> software? That's two different things. Sure, we're not going to fix
>> bugs on pre-4.4.0 releases, but there are pre-4.4.0 clusters running
>> in the wild (yahoo being an example), and they have to be usable with
>> the latest software, otherwise they will never upgrade. I had this in
>> my previous job where they made a big breaking BC change in the
>> metadata between two versions. While it did allow us to delete a lot
>> of code, we ended up supporting two codebases forever, as customers
>> were unable to upgrade to the latest version.
>>
>
> Please check my example above to see if that makes sense to you.
>
>
>>
>> > That is the whole point of having EOL.
>> It's not clear what the point of having EOL is, since we don't have a
>> clearly defined EOL policy nor BC policy.
>>
>
> Please check my example above to see if that makes sense to you.
>
>
>>
>> -Ivan
>>

Re: [DISCUSS] EOL of BookKeeper releases

[Sijie Guo <gu...@gmail.com>]

I think you misunderstood what I was saying.


On Thu, Feb 1, 2018 at 4:53 AM, Ivan Kelly <iv...@apache.org> wrote:

> >> What do we mean by stopping support for BC?
> >
> > It means we will NOT consider BC problems with those old releases.
>
> If EOL means ignoring all BC considerations, then I'm -1 on these
> proposals. But I don't think we need to go to those extremes.
> I think we need to specify what BC guarantees we give on any version,
> and then EOL policies will follow from that. The 3 big BC
> considerations, as I see it, are:
> 1) Data on the bookie
> 2) Client wire protocol
> 3) Data shared between bookies and clients (i.e. zookeeper metadata)
>
> For 1 we can have quite a short timeframe for EOL old software.
> Bookies can be upgraded individually, and this is controlled by the
> administrator.
> For 2 we need a bit longer, because within an organisation there may
> be many different versions of the client knocking around and it's hard
> to get everyone to upgrade.
> For 3 we can pretty much never have a hard break. If a cluster was
> initialized with an old version, they must be able to able to upgrade
> that cluster to the latest version of the software. Otherwise they
> will stick with the old version forever.
>

when I am saying stop all BC consideration, I mean:

1) all the data and metadata will be upgraded to the minimal version we
claim to support.
2) all the client would have to upgrade to the minimal version we claim to
support for wire protocol.

Let's take 4.7.0 as the example, when I am saying enforce EOL at 4.7.0. It
means 4.7.0 will still compatible with 4.1.0. but at 4.7.0 we are thinking
about dropping 4.1.x, 4.2.x and 4.3.x.

so at 4.7.0:

- we will have to develop a tool to upgrade metadata and data to the
version that will be recognized by 4.4.x and onwards releases.

so at 4.8.0:

- we will not consider clients older than 4.4.x. all the legal code can be
potentially removed.

That means if a cluster (either metadata/data produced by clients older
than 4.4.x or still have old clients) want to upgrade to 4.8.0, it has to
first upgrade the cluster (both metadata and data) to 4.7.0 following the
instructions provided at 4.7.0
and upgrade all the clients to minimal version that 4.8.0 will support.
After all is done first, the cluster can be upgraded to 4.8.0.


Hope this example make things clear.

Enforcing EOL means dropping the BC considerations between the oldest
version and newest version and limit the BC considerations within a limited
number of versions which is more controllable, maintenanable. Using the
example above,
it means if you want to upgrade from 4.7.x to 4.8.x, please use the tools
provided in 4.7.x to ensure metadata/data are upgraded to latest version
and no more clients older than 4.4.0 talk to the cluster.


>
> > for example, the password and digest type was introduced at a very old
> > release.
> > There is code assuming ledgers might not have password or digest.
> However,
> > the releases onwards would already have those fields.
> > If we stop supporting those old version, we can remove that special BC
> > code, which will make code clearer to maintain.
>
> This code is in the client. I'm fine with stopping support for all
> clients older than a certain era. However, yahoo is still on 4.3, so
> we should at least support back that far until they've been migrated
> forward.
>

the yahoo branch is between 4.3 and 4.4. it is a special case that we need
to take care of. we are not dropping it out of consideration radar.


>
> >> The BC
> >> tests should help in this regard, but we need keep testing scenarios
> >> where the original cluster was brought up in the oldest version
> >> (4.1.0).
> >
> > If we stop supporting 4.1.0, we don't need to test those scenarios. We
> > started with from the lowest version we are claiming to support.
>
> Are we supporting the software or data that was created with that
> software? That's two different things. Sure, we're not going to fix
> bugs on pre-4.4.0 releases, but there are pre-4.4.0 clusters running
> in the wild (yahoo being an example), and they have to be usable with
> the latest software, otherwise they will never upgrade. I had this in
> my previous job where they made a big breaking BC change in the
> metadata between two versions. While it did allow us to delete a lot
> of code, we ended up supporting two codebases forever, as customers
> were unable to upgrade to the latest version.
>

Please check my example above to see if that makes sense to you.


>
> > That is the whole point of having EOL.
> It's not clear what the point of having EOL is, since we don't have a
> clearly defined EOL policy nor BC policy.
>

Please check my example above to see if that makes sense to you.


>
> -Ivan
>

Re: [DISCUSS] EOL of BookKeeper releases

[Ivan Kelly <iv...@apache.org>]

>> What do we mean by stopping support for BC?
>
> It means we will NOT consider BC problems with those old releases.

If EOL means ignoring all BC considerations, then I'm -1 on these
proposals. But I don't think we need to go to those extremes.
I think we need to specify what BC guarantees we give on any version,
and then EOL policies will follow from that. The 3 big BC
considerations, as I see it, are:
1) Data on the bookie
2) Client wire protocol
3) Data shared between bookies and clients (i.e. zookeeper metadata)

For 1 we can have quite a short timeframe for EOL old software.
Bookies can be upgraded individually, and this is controlled by the
administrator.
For 2 we need a bit longer, because within an organisation there may
be many different versions of the client knocking around and it's hard
to get everyone to upgrade.
For 3 we can pretty much never have a hard break. If a cluster was
initialized with an old version, they must be able to able to upgrade
that cluster to the latest version of the software. Otherwise they
will stick with the old version forever.

> for example, the password and digest type was introduced at a very old
> release.
> There is code assuming ledgers might not have password or digest. However,
> the releases onwards would already have those fields.
> If we stop supporting those old version, we can remove that special BC
> code, which will make code clearer to maintain.

This code is in the client. I'm fine with stopping support for all
clients older than a certain era. However, yahoo is still on 4.3, so
we should at least support back that far until they've been migrated
forward.

>> The BC
>> tests should help in this regard, but we need keep testing scenarios
>> where the original cluster was brought up in the oldest version
>> (4.1.0).
>
> If we stop supporting 4.1.0, we don't need to test those scenarios. We
> started with from the lowest version we are claiming to support.

Are we supporting the software or data that was created with that
software? That's two different things. Sure, we're not going to fix
bugs on pre-4.4.0 releases, but there are pre-4.4.0 clusters running
in the wild (yahoo being an example), and they have to be usable with
the latest software, otherwise they will never upgrade. I had this in
my previous job where they made a big breaking BC change in the
metadata between two versions. While it did allow us to delete a lot
of code, we ended up supporting two codebases forever, as customers
were unable to upgrade to the latest version.

> That is the whole point of having EOL.
It's not clear what the point of having EOL is, since we don't have a
clearly defined EOL policy nor BC policy.

-Ivan

Re: [DISCUSS] EOL of BookKeeper releases

[Sijie Guo <gu...@gmail.com>]

On Thu, Feb 1, 2018 at 2:51 AM, Ivan Kelly <iv...@apache.org> wrote:

> > We have switched to Time Based Release Schedule
> > <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> BP-13+-+Time+Based+Release+Plan>
> > since
> > 4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
> > 4.6.1. And the 4.7.0 release is coming in a month or so. According to
> > BP-13, we need to enforce an EOL policy for our releases: keep the
> releases
> > in one year and mark the releases older than one year as EOL, no BC would
> > be considered with those versions.
> Perhaps we should define an EOL policy before EOLing releases. As part
> of this, we should define what we support for BC and what we don't.
> For example, do we support BC on all ledger managers? Just
> hierarchical and flat?
>
> > I would like to propose we start enforcing EOL at 4.7.0 release:
> +1
>
> > - we only keep all the releases higher than 4.4.0 (4.4.0 was released
> about
> > one year and half ago).
> inclusive or exclusive of 4.4.0?
>

Inclusive.

>
> > - stop supporting BC with releases smaller than 4.4.0. This includes:
> move
> > the documentation of those releases as archived, remove them from BC
> tests,
> > delete the legacy code that was there for supporting BC with these
> releases.
> What do we mean by stopping support for BC?


It means we will NOT consider BC problems with those old releases.


> We haven't broken client
> compatibility since 4.1.0. I don't think we have any special code for
> handling it. But I think there is special code for handling old ledger
> metadata formats (which haven't been upgrades). If someone started
> their cluster on version 4.1.0, and upgraded version by version to
> 4.6, then the ledger metadata from 4.1.0 will still be there, and
> their 4.6.0 client will be depending on the legacy code paths to
> function.

I don't think it's as simple as just removing the legacy
> code, and it's something we need to be very careful about.


It is case by case.

for example, the password and digest type was introduced at a very old
release.
There is code assuming ledgers might not have password or digest. However,
the releases onwards would already have those fields.
If we stop supporting those old version, we can remove that special BC
code, which will make code clearer to maintain.


> The BC
> tests should help in this regard, but we need keep testing scenarios
> where the original cluster was brought up in the oldest version
> (4.1.0).
>

If we stop supporting 4.1.0, we don't need to test those scenarios. We
started with from the lowest version we are claiming to support.
That is the whole point of having EOL.


>
> -Ivan
>

Re: [DISCUSS] EOL of BookKeeper releases

[Ivan Kelly <iv...@apache.org>]

> We have switched to Time Based Release Schedule
> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-13+-+Time+Based+Release+Plan>
> since
> 4.6.0 and we have 3 successful releases since then - 4.5.1, 4.6.0 and
> 4.6.1. And the 4.7.0 release is coming in a month or so. According to
> BP-13, we need to enforce an EOL policy for our releases: keep the releases
> in one year and mark the releases older than one year as EOL, no BC would
> be considered with those versions.
Perhaps we should define an EOL policy before EOLing releases. As part
of this, we should define what we support for BC and what we don't.
For example, do we support BC on all ledger managers? Just
hierarchical and flat?

> I would like to propose we start enforcing EOL at 4.7.0 release:
+1

> - we only keep all the releases higher than 4.4.0 (4.4.0 was released about
> one year and half ago).
inclusive or exclusive of 4.4.0?

> - stop supporting BC with releases smaller than 4.4.0. This includes: move
> the documentation of those releases as archived, remove them from BC tests,
> delete the legacy code that was there for supporting BC with these releases.
What do we mean by stopping support for BC? We haven't broken client
compatibility since 4.1.0. I don't think we have any special code for
handling it. But I think there is special code for handling old ledger
metadata formats (which haven't been upgrades). If someone started
their cluster on version 4.1.0, and upgraded version by version to
4.6, then the ledger metadata from 4.1.0 will still be there, and
their 4.6.0 client will be depending on the legacy code paths to
function. I don't think it's as simple as just removing the legacy
code, and it's something we need to be very careful about. The BC
tests should help in this regard, but we need keep testing scenarios
where the original cluster was brought up in the oldest version
(4.1.0).

-Ivan