You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Becket Qin <be...@gmail.com> on 2023/06/11 12:30:55 UTC

[DISCUSS] FLIP-321: Introduce an API deprecation process

Hi folks,

As one of the release 2.0 efforts, the release managers were discussing our
API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
are relevant to this topic. These two FLIPs defined the stability guarantee
of the programming APIs with various different stability annotations, and
the promotion process. A recap of the conclusion is following:

Stability:
@Internal API: can change between major/minor/patch releases.
@Experimental API: can change between major/minor/patch releases.
@PublicEvolving API: can change between major/minor releases.
@Public API: can only change between major releases.

Promotion:
An @Experimental API should be promoted to @PublicEvolving after two
releases, and a @PublicEvolving API should be promoted to @Public API after
two releases, unless there is a documented reason not to do so.

One thing not mentioned in these two FLIPs is the API deprecation process,
which is in fact critical and fundamental to how the stability guarantee is
provided in practice, because the stability is all about removing existing
APIs. For example, if we want to change a method "ResultFoo foo(ArgumentFoo
arg)" to "ResultBar bar(ArgumentBar arg)", there will be two ways to do
this:

1. Mark method "foo" as deprecated and add the new method "bar". At some
point later, remove the method "foo".
2. Simply change the API in place, that basically means removing method foo
and adding method bar at the same time.

In the first option, users are given a period with stability guarantee to
migrate from "foo" to "bar". For the second option, this migration period
is effectively 0. A zero migration period is problematic because end users
may need a feature/bug fix from a new version, but cannot upgrade right
away due to some backwards compatible changes, even though these changes
perfectly comply with the API stability guarantees defined above. So the
migration period is critical to the API stability guarantees for the end
users.

The migration period is essentially how long a deprecated API can be
removed from the source code. So with this FLIP, I'd like to kick off the
discussion about our deprecation process.

https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process

Comments are welcome!

Thanks,

Jiangjie (Becket) Qin

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Chesnay and Stefan,

Thanks for the feedback and sorry for the confusion about Public API
deprecation. I just noticed that there was a mistake in the NOTES part for
Public API due to a copy-paste error... I just fixed it.

To clarify on the deprecation of "Public" APIs.

The FLIP does not propose to remove the code of "Public" APIs with a minor
version bump. The Public APIs still can only be removed with a major
version bump. What this FLIP enforces is the minimum migration period in
addition to this rule.

For example, if the next version after 1.19 is 2.0, without the migration
period specified in this FLIP (that is 2 minor versions), we can
technically remove a Public API from 1.19 right away because the next
release changes the major version. However, this leaves no time for the
users to migrate. With this FLIP, we have to keep the deprecated API for at
least 2 minor releases, so we have to keep it in at least 2.0 and 2.1 due
to this FLIP, and also all the 2.x versions because removing a Public API
requires a major version bump. Therefore, that deprecated Public API can
only be removed in 3.0.

As another example, if after 1.19, we deprecate a Public API in 1.20, and
1.21, then we release 2.0. In this case, the deprecated Public API has been
kept for 2 minor releases (1.20 and 1.21), we can remove it in 2.0 if we
want to. We can also choose to keep it there for longer if we want to. But
if we decide to keep the deprecated API in 2.0, we can only remove it in
3.0.

The "carry recent Public APIs forward into the next major release" thing
> seems to presume a linear release history (aka, if 2.0 is released after
> 1.20, then there will be no 1.21), which I doubt will be the case. The
> idea behind it is good, but I'd say the right conclusion would be to not
> make that API public if we know a new major release hits in 3 months and
> is about to modify it. With a regular schedule for major releases this
> wouldn't be difficult to do.


I agree if we know a brand new Public API (not a replacement for deprecated
API) will be changed in 3 months, it would not make sense to introduce that
in the current major version in the first place. But that is a different
discussion from the deprecation process. The deprecation process is mainly
for the API that is already there. In the other case, if we are talking
about a new API as a replacement for a deprecated API, we have to introduce
it in the current major version, so that users can migrate from the
deprecated API to the new replacement API in the current major version,
which allows them to migrate to the next major version, which only has the
new API, once it is released.

Regarding the linear v.s. non-linear process, if we can enforce the
migration period, I wonder if we still need to release 1.X after 2.x is
released. The idea behind this is that if we can make the major version
upgrade experience smooth for the users, we don't need to release 1.x
anymore after 2.x is released. Instead we can focus on evolving 2.x. With a
decent migration period, users should be able to upgrade to 2.x from 1.x
because they have been given time to migrate away from all the deprecated
APIs in 1.x before 2.0 is released, that is at least 2 minor releases for
Public APIs, 1 minor release for PublicEvolving APIs and 1 patch release
for Experimental APIs.

It would be valuable if we can avoid releasing minor versions for previous
major versions. Releasing a minor version for old major versions usually
means we are backporting some feature to the previous major version, which
likely introduces non-trivial burdens. If users want new features, they
should upgrade to 2.x.

Thanks,

Jiangjie (Becket) Qin


On Tue, Jun 13, 2023 at 10:24 PM Chesnay Schepler <ch...@apache.org>
wrote:

> On 13/06/2023 12:50, Jing Ge wrote:
> > One major issue we have, afaiu, is caused by the lack of
> housekeeping/house
> > cleaning, there are many APIs that were marked as deprecated a few years
> > ago and still don't get removed. Some APIs should be easy to remove and
> > others will need some more clear rules, like the issue discussed at [1].
>
> This is by design. Most of these are @Public APIs that we had to carry
> around until Flink 2.0, because that was the initial guarantee that we
> gave people.
>
>
> As for the FLIP, I like the idea of explicitly writing down a
> deprecation period for APIs, particularly PublicEvolving ones.
> For Experimental I don't think it'd be a problem if we could change them
> right away,
> but looking back a bit I don't think it hurts us to also enforce some
> deprecation period.
> 1 release for both of these sound fine to me.
>
>
> My major point of contention is the removal of Public APIs between minor
> versions.
> This to me would a major setback towards a simpler upgrade path for users.
> If these can be removed in minor versions than what even is a major
> release?
> The very definition we have for Public APIs is that they stick around
> until the next major one.
> Any rule that theoretically allows for breaking changes in Public API in
> every minor release is in my opinion not a viable option.
>
>
> The "carry recent Public APIs forward into the next major release" thing
> seems to presume a linear release history (aka, if 2.0 is released after
> 1.20, then there will be no 1.21), which I doubt will be the case. The
> idea behind it is good, but I'd say the right conclusion would be to not
> make that API public if we know a new major release hits in 3 months and
> is about to modify it. With a regular schedule for major releases this
> wouldn't be difficult to do.
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by John Roesler <vv...@apache.org>.
Hi Xintong,

Thanks for the update!

FWIW, I'm also in agreement with the points that you and Becket agree on. As you can probably guess, I'd also vote for option (2) on how to handle removal of the DataStream API.

This is actually similar to how Kafka handled the removal of Zookeeper, and it's working out quite nicely:
* Introduce a "bridge release" that adds support for KRaft while deprecating Zookeeper
** It was "early access" in 2.8, preview in 3.0, and becoming progressively production-ready in the 3.x series.
** ZK will be deprecated in the 3.x series once KRaft is production ready
* We plan to remove Zookeeper in 4.0

My perspective is that a similar pair of releases to first deprecate DataStream while introducing its replacement and then remove DataStream would actually be good, as it lets us celebrate both milestones in a high profile way and build excitement about Flink moving into a new era. And if we observe our new API stability processes while we do it, it also ensures a positive perception from users, that we're "improving the project" and not just "changing everything". Circling back to the Kafka 3.0/4.0 bridge example, there's a considerable amount of excitement in the community and user base around this change and these releases, and the sentiment I've heard around the process is overwhelmingly positive, to which I think we can attribute in large part the predictability and respectfulness of the formal process.

And with that, I'll keep quiet and let others speak :)

Thanks again for the discussion,
-John

On 2023/06/20 12:43:11 Xintong Song wrote:
> Becket and I had an offline voice call earlier today. We have reached
> consensus on some of the arguments, but still hold different opinions on
> some others. I'd like to post the outcome here for transparency.
> 
> We both agree that:
> 
>    1. Providing a migration period would be beneficial for our users, and
>    we should do that
>    2. Dropping deprecated Public APIs in minor releases, or demoting APIs
>    from Public to PublicEvolving / Experimental / Retired in any version bump,
>    are not good practices.
>    3. Ideally, with this FLIP, developers should be more careful and plan
>    API changes ahead. That means:
>       1. Be more careful with designing APIs and promoting them to Public,
>       so that they won't be changed / removed very soon, and also the
>       maintenance overhead for keeping them after deprecation should be
>       affordable. (I believe this also aligns with John's opinion.)
>       2. Plan the deprecation / removal of old APIs that we don't want to
>       carry to the next major version earlier, so that they'll have the
>       2-minor-release migration period before the next major version bump.
>    4. A practical situation is that, we are not ready for deprecating the
>    DataStream API, nor afford to carry it for another couple of years.
>       1. We cannot deprecate it now, because the new ProcessFunction API
>       (the planned replacement for DataStream API) is not yet ready. We haven't
>       planned the ProcessFunction API earlier because the original
> plan was to do
>       in-place changes directly on the DataStream API in release 2.0, until the
>       smooth migration period is proposed. Before this, the only guarantees we
>       provide for Public APIs is that they'll stay compatible until the next
>       major version, which led to the understanding the in-place changes on
>       Public APIs should be fine.
>       2. We also cannot afford carrying the DataStream API for another
>       couple of years. One of the reasons that we want to refactor / replace
>       DataStream API is that it exposes and depends on too many
> Flink's internal
>       runtime implementations, which has already limited / blocked plans for
>       Flink's internal improvements for many times. Also due to the wide and
>       transitive dependencies on the internals, maintaining two sets of lower
>       APIs itself is expensive and would be better to keep it short. This is
>       admittedly a historical design flaw which should be avoided in future.
> 
> What we haven't reached consensus is about how to deal with the current
> situation around the DataStream API. There are two options.
> 
> 
>    1. We say this FLIP is enforced starting release 2.0. For current 1.x
>    APIs, we provide a migration period with best effort, while allowing
>    exceptions for immediate removal in 2.0. That means we will still try with
>    best effort to get the ProcessFuncion API ready and deprecate the
>    DataStream API in 1.x, but will also be allowed to remove DataStream API in
>    2.0 if it's not deprecated 2 minor releases before the major version bump.
>    2. We strictly follow the process in this FLIP, and will quickly bump
>    the major version from 2.x to 3.0 once the migration period for DataStream
>    API is reached.
> 
> I'm personally in favor of option 1. As the migration-period rule is newly
> proposed, I think it's fair to make exceptions for cases where we already
> missed the best chance for planning the deprecation. Moreover, I do believe
> having a quick major version bump does confuse users. Yes, we can explain
> to users that bumping from 2.x to 3.0 does not cost anything other than the
> removal of an deprecated API. But having to explain this itself is an
> indicator that it might be confusing for users.
> 
> 
> Becket, on the other hand, prefers option 2. From my understanding, his
> major point is that a quick major version bump causes barely any actual
> lose on users, while in option 1 not providing the migration period or
> providing a shorter on is an actual lose on users. (@Becket, please correct
> me if I'm mistaken.)
> 
> 
> And we'd like to hear more feedback from the community.
> 
> 
> Best,
> 
> Xintong
> 
> 
> 
> On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vv...@apache.org> wrote:
> 
> > Hi Becket and Xintong,
> >
> > I hope you don't mind if I chime in a little.
> >
> > Once an API is marked Public, we're committing to support it until it's
> > deprecated, and once it's deprecated, to leave it in place for at least two
> > minor releases and then only remove it in the following major release.
> >
> > As a user, I would be dismayed to discover that the project maintainers
> > demoted a Public API to PublicEvolving or Experimental in order to violate
> > the spirit of the deprecation period for Public APIs. Likewise, I'd view a
> > rapid sequence of minor releases for the purpose of dropping deprecated
> > APIs in the next major release as another violation of the spirit of our
> > guarantees. I'm reminded of this xkcd: https://xkcd.com/1494/
> >
> > I'm glad we're gaming out these situations before we institute this FLIP,
> > and I hope we all converge on a clear understanding of what we're
> > guaranteeing. From where I'm sitting, the essence of this FLIP is
> > specifically to prevent us from doing whatever we want and instead to
> > commit to providing a suitable notice period to users who rely on Public
> > APIs.
> >
> > It honestly shouldn't matter what the maintenance overhead is (or, rather,
> > evaluating the maintenance overhead should be a consideration before we
> > promote an API to Public to begin with). My experience in other projects is
> > that regretting a feature so much you want to drop it asap is extremely
> > rare, and that living with the pain for a little while until the
> > deprecation period expires will probably make us better engineers.
> >
> > With that all said, one thing that is not a violation is to propose a new
> > major release in order to drop especially burdensome deprecated APIs that
> > have already enjoyed their full deprecation period. Those really bad "case
> > A" features will be very rare.
> >
> > Looking over the FLIP again, I see a reference to the release
> > documentation, but not to the build tooling support for the Deprecated
> > annotation, so it might help to share this scenario: The way this will all
> > come together in practice is that, by giving good stability guarantees, we
> > will encourage our users to bump minor versions when they are released,
> > which means that they will see compiler warnings immediately if any of the
> > APIs they've used are deprecated. In fact, I'd recommend for them to use
> > the `-Werror` javac flag to be sure that they don't miss those warnings.
> > Given our release cadence, this means that they will have about six months
> > to migrate off of any deprecated APIs without being forced to avoid
> > updates. And they really need to be able to adopt further minor updates in
> > the series, since they may contain further improvements that actually
> > facilitate migrating off of the deprecated APIs.
> >
> > The compiler warnings are a far more reliable mechanism to advertise
> > deprecation than release notes, since users' own builds will notify them
> > right away about only the deprecations that are relevant to them, but it
> > does rely on users feeling ok to bump up minor versions in a timely
> > fashion. If we consistently burn them by not observing the stability
> > guarantees, they'll try to avoid updates instead, and the whole thing falls
> > apart.
> >
> > Thanks again,
> > -John
> >
> > On 2023/06/19 10:43:05 Becket Qin wrote:
> > > Hi Xintong,
> > >
> > > Let's compare the following cases:
> > >
> > > A. If the maintenance overhead of the deprecated API is high, and we want
> > > to remove it after two minor releases. Then there are two options:
> > >     A1: Demote the API in the major version bump and remove the code
> > with a
> > > minor version bump.
> > >     A2: Do exactly the same as 1-A, except that when removing the code,
> > > bump up the major version. So this might be a short major version.
> > >
> > > B. If the maintenance overhead of the deprecated API is not high,
> > therefore
> > > keeping it for a long time is affordable. There are also two options:
> > >     B1: Same as A-1, demote the API in the major version bump and remove
> > > the code with a minor version bump.
> > >     B2: Keep the API for all the minor versions in the major version, and
> > > only remove the code in the next major version.
> > >
> > > For case B, do we agree that B2 is the way to go?
> > >
> > > For case A:
> > > The following stays the same for A1 and A2:
> > >   - users will lose the API after two minor leases. So the migration
> > period
> > > is the same for A-1 and A-2.
> > >   - A1 and A2 will have exactly the same code after the removal of
> > > deprecated API, the only difference is versioning.
> > >   - To move forward, users need to move to the next minor version in A1,
> > or
> > > to the next major version in A2. Because the code are the same, the
> > actual
> > > effort to upgrade is the same for A1 and A2.
> > >
> > > The differences between A-1 and A-2 are:
> > >   - A1 allows keeping the major version release cadence. A-2 will have a
> > > short major version release.
> > >   - A1 breaks the well understood API semantic, while A-2 does not.
> > >
> > > From what I see, since there is no well establish standard regarding how
> > > long a major version should be released, A short major version release is
> > > potential and emotional. It is not ideal but does not have material
> > > downsides compared with A1. I did not hear anyone complaining about Kafka
> > > only has two 1.x release. However, A1 actually breaks the well understood
> > > API semantic, which has more material impact.
> > >
> > > Also, I'd imagine 90% or more of the Public APIs should fall into case B.
> > > So, short major versions should be very occasional. I'd be very concerned
> > > if the reason we choose A1 is simply because we cannot afford
> > maintaining a
> > > bunch of deprecated APIs until the next major version. This indicates
> > that
> > > the actual problem we need to solve is to lower the maintenance overhead
> > of
> > > deprecated APIs, so that we are comfortable to keep them longer. As John
> > > and I mentioned earlier, there are ways to achieve this and we need to
> > > learn how to do it in Flink. Otherwise, our discussion about versioning
> > > here does not bring much value, because we will end up with a bunch of
> > > short-lived APIs which upset our users, no matter how we version the
> > > releases.
> > >
> > > So, if there are concrete examples that you think will block us from
> > > keeping API stability with affordable cost, let's take a look together
> > and
> > > see if that can be improved.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > > From what I see,
> > >
> > >
> > > On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > > >
> > > > > The part I don't understand is if we are willing to have a migration
> > > > > period, and do a minor version bump to remove an API, what do we
> > lose to
> > > > do
> > > > > a major version bump instead, so we don't break the common versioning
> > > > > semantic?
> > > > >
> > > >
> > > > I think we are talking about the cases where a major version bump
> > happens
> > > > before a deprecated Public API reaches its migration period. So
> > removing
> > > > the API with another major version bump means we have two consecutive
> > major
> > > > versions in a very short time. And as previously mentioned, having
> > frequent
> > > > major version bumps would weaken the value of the commitment "Public
> > API
> > > > stay compatible within a major version". I think users also have
> > > > expectations about how long a major version should live, which should
> > be at
> > > > least a couple of years rather than 1-2 minor releases.
> > > >
> > > > This is another option, but I think it is very likely more expensive
> > than
> > > > > simply bumping the major version. And I imagine there will be
> > questions
> > > > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > > > >
> > > >
> > > > Yes, it's more expensive than simply bumping the major version, but
> > IMHO
> > > > it's probably cheaper than carrying the API until the next major
> > version if
> > > > we don't bump the major version very soon. And see my reply above on
> > why
> > > > not bumping immediately.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <be...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Xintong,
> > > > >
> > > > > Please see the replies below.
> > > > >
> > > > > I see your point, that users would feel surprised if they find
> > things no
> > > > > > longer work when upgrading to another 2.x minor release. However,
> > I'd
> > > > > like
> > > > > > to point out that PublicEvolving APIs would have the similar
> > problem
> > > > > > anyway. So the question is, how do we catch users' attention and
> > make
> > > > > sure
> > > > > > they are aware that the Public APIs in 1.x may no longer be Public
> > in
> > > > > 2.0.
> > > > > > There're various ways to do that, e.g., release notes, warnings in
> > > > logs,
> > > > > > etc.
> > > > >
> > > > > First of all, I am not a fan of removing PublicEvolving APIs in minor
> > > > > version changes. Personally speaking, I want them to be also removed
> > in
> > > > > major version changes.
> > > > >
> > > > > Empirically, I rarely see projects with complex API semantic work
> > well.
> > > > > Many, if not most, users don't read docs / release notes / warnings
> > in
> > > > > logs. I don't think that is their fault. Typically an application
> > > > developer
> > > > > will have to deal with dozens of libraries maintained by various
> > > > > communities / groups. It is just too difficult for them to keep
> > track of
> > > > > all the specific semantics each project puts there. If you think
> > about
> > > > it,
> > > > > how many of Flink developers read all the release notes of Guava,
> > Apache
> > > > > Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of
> > them?
> > > > > One would probably try bumping the dependency version and see if
> > there is
> > > > > an exception and solve them case by case. And if it is a minor
> > version
> > > > > bump, one probably does not expect an exception at all. Another
> > example
> > > > is
> > > > > that in Flink we still have so many usage of deprecated methods all
> > over
> > > > > the place, and I strongly doubt everyone knows when the source code
> > of
> > > > > these methods are deprecated and when they should be removed.
> > > > >
> > > > > So, the most important principle of API is simple and intuitive. The
> > > > > versioning semantic is a simple and universally accepted API
> > stability
> > > > > standard. If we ourselves as Flink developers are relying on this
> > for our
> > > > > own dependencies. I don't think we can expect more from our users.
> > > > >
> > > > > Another possible alternative: whenever there's a deprecated Public
> > API
> > > > that
> > > > > > reaches a major version bump before the migration period, and we
> > also
> > > > > don't
> > > > > > want to carry it for all the next major release series, we may
> > consider
> > > > > > releasing more minor releases for the previous major version after
> > the
> > > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump
> > to
> > > > 2.0,
> > > > > > we can release one more 1.20 after 2.0. That should provide users
> > > > another
> > > > > > choice rather than upgrading to 2.0, while satisfying the
> > > > 2-minor-release
> > > > > > migration period.
> > > > >
> > > > > This is another option, but I think it is very likely more expensive
> > than
> > > > > simply bumping the major version. And I imagine there will be
> > questions
> > > > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > > > >
> > > > > I think my major point is, we should not carry APIs deprecated in a
> > > > > > previous major version along all the next major version series. I'd
> > > > like
> > > > > to
> > > > > > try giving users more commitments, i.e. the migration period, as
> > long
> > > > as
> > > > > it
> > > > > > does not prevent us from making breaking changes. If it doesn't
> > work,
> > > > I'd
> > > > > > be in favor of not providing the migration period, but fallback to
> > only
> > > > > > guarantee the compatibility within the major version.
> > > > >
> > > > > The part I don't understand is if we are willing to have a migration
> > > > > period, and do a minor version bump to remove an API, what do we
> > lose to
> > > > do
> > > > > a major version bump instead, so we don't break the common versioning
> > > > > semantic?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > >
> > > > > > > As an end user who only uses Public APIs, if I don't change my
> > code
> > > > at
> > > > > > > all, my expectation is the following:
> > > > > > > 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > with
> > > > > all
> > > > > > > the 2.x versions.
> > > > > > > Actually I think there are some dependency version resolution
> > > > policies
> > > > > > out
> > > > > > > there which picks the highest minor version when the dependencies
> > > > pull
> > > > > in
> > > > > > > multiple minor versions of the same jar, which may be broken if
> > we
> > > > > remove
> > > > > > > the API in minor releases.
> > > > > > >
> > > > > >
> > > > > > I see your point, that users would feel surprised if they find
> > things
> > > > no
> > > > > > longer work when upgrading to another 2.x minor release. However,
> > I'd
> > > > > like
> > > > > > to point out that PublicEvolving APIs would have the similar
> > problem
> > > > > > anyway. So the question is, how do we catch users' attention and
> > make
> > > > > sure
> > > > > > they are aware that the Public APIs in 1.x may no longer be Public
> > in
> > > > > 2.0.
> > > > > > There're various ways to do that, e.g., release notes, warnings in
> > > > logs,
> > > > > > etc.
> > > > > >
> > > > > > Another possible alternative: whenever there's a deprecated Public
> > API
> > > > > that
> > > > > > reaches a major version bump before the migration period, and we
> > also
> > > > > don't
> > > > > > want to carry it for all the next major release series, we may
> > consider
> > > > > > releasing more minor releases for the previous major version after
> > the
> > > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump
> > to
> > > > 2.0,
> > > > > > we can release one more 1.20 after 2.0. That should provide users
> > > > another
> > > > > > choice rather than upgrading to 2.0, while satisfying the
> > > > 2-minor-release
> > > > > > migration period.
> > > > > >
> > > > > > I think my major point is, we should not carry APIs deprecated in a
> > > > > > previous major version along all the next major version series. I'd
> > > > like
> > > > > to
> > > > > > try giving users more commitments, i.e. the migration period, as
> > long
> > > > as
> > > > > it
> > > > > > does not prevent us from making breaking changes. If it doesn't
> > work,
> > > > I'd
> > > > > > be in favor of not providing the migration period, but fallback to
> > only
> > > > > > guarantee the compatibility within the major version.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcephei@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Becket,
> > > > > > >
> > > > > > > Thanks for the reply! I’d like to continue the conversation about
> > > > > > > compatibility outside of this FLIP thread, but for now, I can
> > accept
> > > > > your
> > > > > > > decision. It’s certainly an improvement.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > > John
> > > > > > >
> > > > > > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Completely agree with all you said.
> > > > > > > >
> > > > > > > > Can we consider only dropping deprecated APIs in major releases
> > > > > across
> > > > > > > the
> > > > > > > >> board? I understand that Experimental and PublicEvolving APIs
> > are
> > > > by
> > > > > > > >> definition less stable, but it seems like this should be
> > reflected
> > > > > in
> > > > > > > the
> > > > > > > >> required deprecation period alone. I.e. that we must keep them
> > > > > around
> > > > > > > for
> > > > > > > >> at least zero or one minor release, not that we can drop them
> > in a
> > > > > > > minor or
> > > > > > > >> patch release.
> > > > > > > >
> > > > > > > > Personally speaking, I would love to do this, for exactly the
> > > > reason
> > > > > > you
> > > > > > > > mentioned. However, I did not propose this due to the following
> > > > > > reasons:
> > > > > > > >
> > > > > > > > 1. I am hesitating a little bit about changing the accepted
> > FLIPs
> > > > too
> > > > > > > soon.
> > > > > > > > 2. More importantly, to avoid slowing down our development. At
> > this
> > > > > > > point,
> > > > > > > > Flink still lacks some design / routines to support good API
> > > > > > > evolvability /
> > > > > > > > extensibility. Just like you said, it takes some time to be
> > good at
> > > > > > this.
> > > > > > > > In this case, my concern is that only removing Experimental /
> > > > > > > > PublicEvolving APIs in major version changes may result in too
> > much
> > > > > > > > overhead and dramatically slow down the development of Flink.
> > So, I
> > > > > was
> > > > > > > > thinking that we can start with the current status. Hopefully
> > after
> > > > > we
> > > > > > > are
> > > > > > > > more comfortable with the maintenance overhead of deprecated
> > APIs,
> > > > we
> > > > > > can
> > > > > > > > then have a stronger guarantee for Experimental /
> > PublicEvolving
> > > > > APIs.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > vvcephei@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Becket,
> > > > > > > >>
> > > > > > > >> Thanks for this FLIP! Having a deprecation process is really
> > > > > > important.
> > > > > > > I
> > > > > > > >> understand some people’s concerns about the additional burden
> > for
> > > > > > > project
> > > > > > > >> maintainers, but my personal experience with Kafka has been
> > that
> > > > > it’s
> > > > > > > very
> > > > > > > >> liveable and that it’s well worth the benefit to users. In
> > fact,
> > > > > users
> > > > > > > >> being able to confidently upgrade is also a benefit to
> > > > maintainers,
> > > > > as
> > > > > > > we
> > > > > > > >> will get fewer questions from people stuck on very old
> > versions.
> > > > > > > >>
> > > > > > > >> One question:
> > > > > > > >> Can we consider only dropping deprecated APIs in major
> > releases
> > > > > across
> > > > > > > the
> > > > > > > >> board? I understand that Experimental and PublicEvolving APIs
> > are
> > > > by
> > > > > > > >> definition less stable, but it seems like this should be
> > reflected
> > > > > in
> > > > > > > the
> > > > > > > >> required deprecation period alone. I.e. that we must keep them
> > > > > around
> > > > > > > for
> > > > > > > >> at least zero or one minor release, not that we can drop them
> > in a
> > > > > > > minor or
> > > > > > > >> patch release.
> > > > > > > >>
> > > > > > > >> The advantage of forbidding the removal of any API in minor or
> > > > patch
> > > > > > > >> releases is that users will get a strong guarantee that they
> > can
> > > > > bump
> > > > > > > the
> > > > > > > >> minor or patch version and still be able to compile, or even
> > just
> > > > > > > re-link
> > > > > > > >> and know that they won’t face “MethodDef” exceptions at run
> > time.
> > > > > This
> > > > > > > is a
> > > > > > > >> binary guarantee: if we allow removing  even Experimental APIs
> > > > > outside
> > > > > > > of
> > > > > > > >> major releases, users can no longer confidently upgrade.
> > > > > > > >>
> > > > > > > >> Aside from that, I’d share my 2 cents on a couple of points:
> > > > > > > >> * I’d use the official Deprecated annotation instead of
> > > > introducing
> > > > > > our
> > > > > > > >> own flavor (Retired, etc), since Deprecated is well integrated
> > > > into
> > > > > > > build
> > > > > > > >> tools and IDEs.
> > > > > > > >> * I wouldn’t worry about a demotion process in this FLIP; it
> > seems
> > > > > > > >> orthogonal, and something that should probably be taken
> > > > case-by-case
> > > > > > > >> anyway.
> > > > > > > >> * Aside from deprecation and removal, there have been some
> > > > > discussions
> > > > > > > >> about how to evolve APIs and behavior in compatible ways.
> > This is
> > > > > > > somewhat
> > > > > > > >> of an art, and if folks haven’t wrestled with it before, it’ll
> > > > take
> > > > > > some
> > > > > > > >> time to become good at it. I feel like this topic should also
> > be
> > > > > > > orthogonal
> > > > > > > >> to this FLIP, but FWIW, my suggestion would be to adopt a
> > simple
> > > > > > policy
> > > > > > > not
> > > > > > > >> to break existing user programs, and leave the “how” up to
> > > > > > implementers
> > > > > > > and
> > > > > > > >> reviewers.
> > > > > > > >>
> > > > > > > >> Thanks again,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > >> > Hi All,
> > > > > > > >> >
> > > > > > > >> > The @Public -> @PublicEvolving proposed by Xintong is a
> > great
> > > > > idea.
> > > > > > > >> > Especially, after he suggest @PublicRetired, i.e.
> > > > @PublicEvolving
> > > > > > --(2
> > > > > > > >> > minor release)--> @Public --> @deprecated --(1 major
> > > > > > > >> > release)--> @PublicRetired. It will provide a lot of
> > flexibility
> > > > > > > without
> > > > > > > >> > breaking any rules we had. @Public APIs are allowed to
> > change
> > > > > > between
> > > > > > > >> major
> > > > > > > >> > releases. Changing annotations is acceptable and provides
> > > > > additional
> > > > > > > >> > tolerance i.e. user-friendliness, since the APIs themself
> > are
> > > > not
> > > > > > > >> changed.
> > > > > > > >> >
> > > > > > > >> > I had similar thoughts when I was facing those issues. I
> > want to
> > > > > > move
> > > > > > > one
> > > > > > > >> > step further and suggest introducing one more annotation
> > > > @Retired.
> > > > > > > >> >
> > > > > > > >> > Not like the @PublicRetired which is a compromise of
> > downgrading
> > > > > > > @Public
> > > > > > > >> to
> > > > > > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java
> > > > standard
> > > > > > > >> > @deprecated should be used in the early stage of the
> > deprecation
> > > > > > > process
> > > > > > > >> > and doesn't really meet our requirement. Since Java does not
> > > > allow
> > > > > > us
> > > > > > > to
> > > > > > > >> > extend annotation, I think it would be feasible to have the
> > new
> > > > > > > @Retired
> > > > > > > >> to
> > > > > > > >> > help us monitor and manage the deprecation process, house
> > > > > cleaning,
> > > > > > > etc.
> > > > > > > >> >
> > > > > > > >> > Some ideas could be(open for discussion):
> > > > > > > >> >
> > > > > > > >> > @Retired:
> > > > > > > >> >
> > > > > > > >> > 1. There must be a replacement with functionality
> > compatibility
> > > > > > before
> > > > > > > >> APIs
> > > > > > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes
> > on
> > > > the
> > > > > ML
> > > > > > > are
> > > > > > > >> > mandatory (a FLIP is recommended).
> > > > > > > >> > 2. APIs marked as @Retired will be removed after 1 minor
> > release
> > > > > > > sharply
> > > > > > > >> > (using ArchUnit to force it, needs further check whether it
> > is
> > > > > > > possible).
> > > > > > > >> > Devs who marked them as @Retired are responsible to remove
> > them.
> > > > > > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
> > are
> > > > > > > >> > recommended. @Experimental -> @Retired and @Internal ->
> > @Retired
> > > > > > could
> > > > > > > >> also
> > > > > > > >> > be used if it can increase user-friendliness or
> > > > dev-friendliness,
> > > > > > but
> > > > > > > not
> > > > > > > >> > mandatory.
> > > > > > > >> > 4. Some variables will be defined in @Retired to support the
> > > > > > > deprecation
> > > > > > > >> > process management. Further extension is possible, since the
> > > > > > > annotation
> > > > > > > >> is
> > > > > > > >> > built by us.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Best regards,
> > > > > > > >> > Jing
> > > > > > > >> >
> > > > > > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > becket.qin@gmail.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> >> Hi Xintong,
> > > > > > > >> >>
> > > > > > > >> >> Thanks for the explanation. Please see the replies inline
> > > > below.
> > > > > > > >> >>
> > > > > > > >> >> I agree. And from my understanding, demoting a Public API
> > is
> > > > > also a
> > > > > > > >> kind of
> > > > > > > >> >> > such change, just like removing one, which can only
> > happen
> > > > with
> > > > > > > major
> > > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > > > APIs
> > > > > > > >> anytime,
> > > > > > > >> >> but
> > > > > > > >> >> > only in the case major version bumps happen before
> > reaching
> > > > the
> > > > > > > >> >> > 2-minor-release migration period. Actually, demoting
> > would
> > > > be a
> > > > > > > weaker
> > > > > > > >> >> > change compared to removing the API immediately upon
> > major
> > > > > > version
> > > > > > > >> bumps,
> > > > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > > > migration
> > > > > > > >> >> period.
> > > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving`
> > sounds
> > > > > > > against
> > > > > > > >> >> > conventions, we may introduce a new annotation if
> > necessary,
> > > > > > e.g.,
> > > > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > > > >> >>
> > > > > > > >> >> As an end user who only uses Public APIs, if I don't
> > change my
> > > > > code
> > > > > > > at
> > > > > > > >> all,
> > > > > > > >> >> my expectation is the following:
> > > > > > > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> > fine
> > > > > > with
> > > > > > > all
> > > > > > > >> >> the 2.x versions.
> > > > > > > >> >> Actually I think there are some dependency version
> > resolution
> > > > > > > policies
> > > > > > > >> out
> > > > > > > >> >> there which picks the highest minor version when the
> > > > dependencies
> > > > > > > pull
> > > > > > > >> in
> > > > > > > >> >> multiple minor versions of the same jar, which may be
> > broken if
> > > > > we
> > > > > > > >> remove
> > > > > > > >> >> the API in minor releases.
> > > > > > > >> >>
> > > > > > > >> >> I'm not sure about this. Yes, it's completely "legal" that
> > we
> > > > > bump
> > > > > > up
> > > > > > > >> the
> > > > > > > >> >> > major version whenever a breaking change is needed.
> > However,
> > > > > this
> > > > > > > also
> > > > > > > >> >> > weakens the value of the commitment that public APIs will
> > > > stay
> > > > > > > stable
> > > > > > > >> >> > within the major release series, as the series can end
> > > > anytime.
> > > > > > > IMHO,
> > > > > > > >> >> short
> > > > > > > >> >> > major release series are not something "make the end
> > users
> > > > > > happy",
> > > > > > > but
> > > > > > > >> >> > backdoors that allow us as the developers to make
> > frequent
> > > > > > breaking
> > > > > > > >> >> > changes. On the contrary, with the demoting approach, we
> > can
> > > > > > still
> > > > > > > >> have
> > > > > > > >> >> > longer major release series, while only allowing Public
> > APIs
> > > > > > > >> deprecated
> > > > > > > >> >> at
> > > > > > > >> >> > the end of the previous major version to be removed in
> > the
> > > > next
> > > > > > > major
> > > > > > > >> >> > version.
> > > > > > > >> >>
> > > > > > > >> >> I totally agree that frequent major version bumps are not
> > > > ideal,
> > > > > > but
> > > > > > > >> here
> > > > > > > >> >> we are comparing it with a minor version bump which
> > removes a
> > > > > > Public
> > > > > > > >> API.
> > > > > > > >> >> So the context is that we have already decided to remove
> > this
> > > > > > Public
> > > > > > > API
> > > > > > > >> >> while keeping everything else backwards compatible. I
> > think a
> > > > > major
> > > > > > > >> version
> > > > > > > >> >> bump is a commonly understood signal here, compared with a
> > > > minor
> > > > > > > version
> > > > > > > >> >> change. From end users' perspective, for those who are not
> > > > > > impacted,
> > > > > > > in
> > > > > > > >> >> this case upgrading a major version is not necessarily more
> > > > > > involved
> > > > > > > >> than
> > > > > > > >> >> upgrading a minor version - both should be as smooth as a
> > > > > > dependency
> > > > > > > >> >> version change. For those who are impacted, they will lose
> > the
> > > > > > Public
> > > > > > > >> API
> > > > > > > >> >> anyways and a major version bump ensures there is no
> > surprise.
> > > > > > > >> >>
> > > > > > > >> >> Thanks,
> > > > > > > >> >>
> > > > > > > >> >> Jiangjie (Becket) Qin
> > > > > > > >> >>
> > > > > > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > tonysong820@gmail.com
> > > > > > > >
> > > > > > > >> >> wrote:
> > > > > > > >> >>
> > > > > > > >> >> > Public API is a well defined common concept, and one of
> > its
> > > > > > > >> >> >> convention is that it only changes with a major version
> > > > > change.
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> > I agree. And from my understanding, demoting a Public
> > API is
> > > > > > also a
> > > > > > > >> kind
> > > > > > > >> >> > of such change, just like removing one, which can only
> > happen
> > > > > > with
> > > > > > > >> major
> > > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > > > APIs
> > > > > > > >> anytime,
> > > > > > > >> >> but
> > > > > > > >> >> > only in the case major version bumps happen before
> > reaching
> > > > the
> > > > > > > >> >> > 2-minor-release migration period. Actually, demoting
> > would
> > > > be a
> > > > > > > weaker
> > > > > > > >> >> > change compared to removing the API immediately upon
> > major
> > > > > > version
> > > > > > > >> bumps,
> > > > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > > > migration
> > > > > > > >> >> period.
> > > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving`
> > sounds
> > > > > > > against
> > > > > > > >> >> > conventions, we may introduce a new annotation if
> > necessary,
> > > > > > e.g.,
> > > > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > > > >> >> >
> > > > > > > >> >> > But it should be
> > > > > > > >> >> >> completely OK to bump up the major version if we really
> > want
> > > > > to
> > > > > > > get
> > > > > > > >> rid
> > > > > > > >> >> of
> > > > > > > >> >> >> a public API, right?
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> > I'm not sure about this. Yes, it's completely "legal"
> > that we
> > > > > > bump
> > > > > > > up
> > > > > > > >> the
> > > > > > > >> >> > major version whenever a breaking change is needed.
> > However,
> > > > > this
> > > > > > > also
> > > > > > > >> >> > weakens the value of the commitment that public APIs will
> > > > stay
> > > > > > > stable
> > > > > > > >> >> > within the major release series, as the series can end
> > > > anytime.
> > > > > > > IMHO,
> > > > > > > >> >> short
> > > > > > > >> >> > major release series are not something "make the end
> > users
> > > > > > happy",
> > > > > > > but
> > > > > > > >> >> > backdoors that allow us as the developers to make
> > frequent
> > > > > > breaking
> > > > > > > >> >> > changes. On the contrary, with the demoting approach, we
> > can
> > > > > > still
> > > > > > > >> have
> > > > > > > >> >> > longer major release series, while only allowing Public
> > APIs
> > > > > > > >> deprecated
> > > > > > > >> >> at
> > > > > > > >> >> > the end of the previous major version to be removed in
> > the
> > > > next
> > > > > > > major
> > > > > > > >> >> > version.
> > > > > > > >> >> >
> > > > > > > >> >> > Given our track record I would prefer a regular cycle
> > (1-2
> > > > > years)
> > > > > > > to
> > > > > > > >> >> >> force us to think about this whole topic, and not put it
> > > > again
> > > > > > to
> > > > > > > the
> > > > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > > > when
> > > > > > > >> breaking
> > > > > > > >> >> >> changes can be made.
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> > +1. I personally think 2-3 years would be a good time
> > for new
> > > > > > major
> > > > > > > >> >> > versions, or longer if there's no breaking changes
> > needed.
> > > > That
> > > > > > > makes
> > > > > > > >> 1-2
> > > > > > > >> >> > year a perfect time to revisit the topic, while leaving
> > us
> > > > more
> > > > > > > time
> > > > > > > >> to
> > > > > > > >> >> > prepare the major release if needed.
> > > > > > > >> >> >
> > > > > > > >> >> > Best,
> > > > > > > >> >> >
> > > > > > > >> >> > Xintong
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > > > > chesnay@apache.org
> > > > > > > >> >
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > >> >> >> > It would be valuable if we can avoid releasing minor
> > > > > versions
> > > > > > > for
> > > > > > > >> >> >> previous
> > > > > > > >> >> >> > major versions.
> > > > > > > >> >> >>
> > > > > > > >> >> >> On paper, /absolutely /agree, but I'm not sure how
> > viable
> > > > that
> > > > > > is
> > > > > > > in
> > > > > > > >> >> >> practice.
> > > > > > > >> >> >>
> > > > > > > >> >> >> On the current 2.0 agenda is potentially dropping
> > support
> > > > for
> > > > > > Java
> > > > > > > >> 8/11,
> > > > > > > >> >> >> which may very well be a problem for our current users.
> > > > > > > >> >> >>
> > > > > > > >> >> >>
> > > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > >> >> >> > Thanks for the feedback and sorry for the confusion
> > about
> > > > > > Public
> > > > > > > >> API
> > > > > > > >> >> >> > deprecation. I just noticed that there was a mistake
> > in
> > > > the
> > > > > > > NOTES
> > > > > > > >> part
> > > > > > > >> >> >> for
> > > > > > > >> >> >> > Public API due to a copy-paste error... I just fixed
> > it.
> > > > > > > >> >> >> I'm very relieved to hear that. Glad to hear that we
> > are on
> > > > > the
> > > > > > > same
> > > > > > > >> >> >> page on that note.
> > > > > > > >> >> >>
> > > > > > > >> >> >>
> > > > > > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > >> >> >> > But it should be
> > > > > > > >> >> >> > completely OK to bump up the major version if we
> > really
> > > > want
> > > > > > to
> > > > > > > get
> > > > > > > >> >> rid
> > > > > > > >> >> >> of
> > > > > > > >> >> >> > a public API, right?
> > > > > > > >> >> >>
> > > > > > > >> >> >> Technically yes, but look at how long it took to get us
> > to
> > > > > 2.0.
> > > > > > ;)
> > > > > > > >> >> >>
> > > > > > > >> >> >> There's a separate discussion to be had on the cadence
> > of
> > > > > major
> > > > > > > >> releases
> > > > > > > >> >> >> going forward, and there seem to be different opinions
> > on
> > > > > that.
> > > > > > > >> >> >>
> > > > > > > >> >> >> If we take the Kafka example of 2 minor releases between
> > > > major
> > > > > > > ones,
> > > > > > > >> >> >> that for us means that users have to potentially deal
> > with
> > > > > > > breaking
> > > > > > > >> >> >> changes every 6 months, which seems like a lot.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Given our track record I would prefer a regular cycle
> > (1-2
> > > > > > years)
> > > > > > > to
> > > > > > > >> >> >> force us to think about this whole topic, and not put it
> > > > again
> > > > > > to
> > > > > > > the
> > > > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > > > when
> > > > > > > >> breaking
> > > > > > > >> >> >> changes can be made.
> > > > > > > >> >> >>
> > > > > > > >> >> >> But again, maybe this should be in a separate thread.
> > > > > > > >> >> >>
> > > > > > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > >> >> >> > Do you have an example of behavioral change in mind?
> > Not
> > > > > sure
> > > > > > I
> > > > > > > >> fully
> > > > > > > >> >> >> > understand the concern for behavioral change here.
> > > > > > > >> >> >>
> > > > > > > >> >> >> This could be a lot of things. It can be performance in
> > > > > certain
> > > > > > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly)
> > relied
> > > > > upon
> > > > > > > >> >> >> (https://xkcd.com/1172/), a semantic change to some
> > API.
> > > > > > > >> >> >>
> > > > > > > >> >> >> For a concrete example, consider the job submission. A
> > few
> > > > > > > releases
> > > > > > > >> back
> > > > > > > >> >> >> we made changes such that the initialization of the job
> > > > master
> > > > > > > >> happens
> > > > > > > >> >> >> asynchronously.
> > > > > > > >> >> >> This meant the job submission call returns sooner, and
> > the
> > > > job
> > > > > > > state
> > > > > > > >> >> >> enum was extended to cover this state.
> > > > > > > >> >> >> API-wise we consider this a compatible change, but the
> > > > > observed
> > > > > > > >> behavior
> > > > > > > >> >> >> may be different.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Metrics are another example; I believe over time we
> > changed
> > > > > what
> > > > > > > some
> > > > > > > >> >> >> metrics returned a few times.
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Martijn,

Thanks for the reply. Regarding the behavioral stability guarantee, by
definition, an "API" always consists of the signature + behavior. So, the
behavior is a native part of an API. Therefore, behavioral changes should
be treated with the same guarantee as signature changes.

+1 on having toolings to enforce the conventions.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jun 28, 2023 at 5:09 AM Martijn Visser <ma...@apache.org>
wrote:

> Hi all,
>
> Thanks for the lively and good discussion. Given the length of the
> discussion, I skimmed through and then did a deep dive on the latest state
> of the FLIP. I think the FLIP is overall in a good state and ready to bring
> to a vote.
>
> One thing that I did notice while skimming through the discussions is that
> I think there are some follow-ups that could be worthy of a FLIP and a
> discussion. For example, if/what/how does the Flink community offer any
> behavioral stability guarantees, or other types of expectations and
> guarantees. I also do think that we must have tooling in place to implement
> this FLIP (and also FLIP-196 and FLIP-197), to avoid that we're not
> creating a policy on paper, but also have the means to enforce it. Last but
> not least, I echo Jark's point that we can't estimate maintenance cost with
> a concrete design and code/POC. For me, that means that a contributor can
> choose to propose a deviation, but that the contributor would need to
> explicitly mention it in the FLIP and get it discussed/voted on as part of
> the FLIP process. But the starting point is as defined in this and other
> relevant FLIPs.
>
> Best regards,
>
> Martijn
>
> On Tue, Jun 27, 2023 at 3:38 AM Becket Qin <be...@gmail.com> wrote:
>
> > Hi Xintong, Jark and Jing,
> >
> > Thanks for the reply. Yes, we can only mark the DataStream API as
> > @Deprecated after the ProcessFunction API is fully functional and mature.
> >
> > It is a fair point that the condition of marking a @Public API as
> > deprecated should also be a part of this FLIP. I just added that to the
> > FLIP wiki. This is probably more of a clarification on the existing
> > convention, rather than a change.
> >
> > It looks like we are on the same page now for this FLIP. If so, I'll
> start
> > a VOTE thread in two days.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Jun 26, 2023 at 8:09 PM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > >
> > > > Considering DataStream API is the most fundamental and complex API of
> > > > Flink, I think it is worth a longer time than the general process for
> > the
> > > > deprecation period to wait for the new API be mature.
> > > >
> > >
> > > This inspires me. In this specific case, compared to how long should
> > > DataStream API be removed after deprecation, it's probably more
> important
> > > to answer the question how long would ProcessFunction API become mature
> > and
> > > stable after being introduced. According to FLIP-197[1], it requires 4
> > > minor releases by default to promote an @Experimental API to @Public.
> And
> > > for ProcessFunction API, which aims to replace DataStream API as one of
> > the
> > > most fundamental API of Flink, I'd expect this to take at least the
> > default
> > > time, or even longer. And we probably should wait until we believe
> > > ProcessFunction API is stable to mark DataStream API as deprecated,
> > rather
> > > than as soon as it's introduced. Assuming we introduce the
> > ProcessFunction
> > > API in 2.0, that means we would need to wait for 6 minor releases (4
> for
> > > the new API to become stable, and 2 for the migration period) to remove
> > > DataStream API, which is ~2.5 year (assuming 5 months / minor release),
> > > which sounds acceptable for another major version bump.
> > >
> > > To wrap things up, it seems to me, sadly, that anyway we cannot avoid
> the
> > > overhead for maintaining both DataStream & ProcessFunction APIs for at
> > > least 6 minor releases.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > >
> > >
> > >
> > > On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just want to make sure we are on the same page. There is another
> > > example[1]
> > > > I was aware of recently that shows why more factors need to be taken
> > care
> > > > of than just the migration period. Thanks Galen for your hint.
> > > >
> > > > To put it simply, the concern about API deprecation is not that
> > > deprecated
> > > > APIs have been removed too early (min migration period is required).
> > The
> > > > major concern is that APIs are marked as deprecated for a (too) long
> > > time,
> > > > much longer than the migration period discussed in this thread,
> afaik.
> > > > Since there is no clear picture/definition, no one knows when to do
> the
> > > > migration for users(after the migration period has expired) and when
> to
> > > > remove deprecated APIs for Flink developers.
> > > >
> > > > Based on all the information I knew, there are two kinds of obstacles
> > > that
> > > > will and should block the deprecation process:
> > > >
> > > > 1. Lack of functionalities in new APIs. It happens e.g. with the
> > > > SourceFunction to FLIP-27 Source migration. Users who rely on those
> > > > functions can not migrate to new APIs.
> > > > 2. new APIs have critical bugs. An example could be found at [1].
> Users
> > > > have to stick to the deprecated APIs.
> > > >
> > > > Since FLIP-321 is focusing on the API deprecation process, those
> > blocking
> > > > issues deserve attention and should be put into the FLIP. The current
> > > FLIP
> > > > seems to only focus on migration periods. If we consider those
> blocking
> > > > issues as orthogonal issues that are beyond the scope of this
> > discussion,
> > > > does it make sense to change the FLIP title to something like
> > "Introduce
> > > > minimum migration periods of API deprecation process"?
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
> > > >
> > > > On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > > I agree with Jingsong and Becket.
> > > > >
> > > > > Look at the legacy SourceFunction (a small part of DataStream API),
> > > > > the SourceFunction is still not and can't be marked deprecated[1]
> > until
> > > > > now after the new Source was released 2 years ago, because the new
> > > Source
> > > > > still can't fully consume the abilities of legacy API. Considering
> > > > > DataStream
> > > > > API is the most fundamental and complex API of Flink, I think it is
> > > worth
> > > > > a longer time than the general process for the deprecation period
> to
> > > > > wait for the new API be mature. The above 2 options sound a bit of
> > rush
> > > > > for such a widely used API.
> > > > >
> > > > > I fully understand the concern of maintenance overhead, but it's a
> > bit
> > > > hard
> > > > > for others to estimate maintenance costs without a concrete design
> > and
> > > > code
> > > > > of the new ProcessFunction API. I agree with Becket that maybe we
> can
> > > > > re-evaluate the API deprecation process once we have the new
> > > > > ProcessFunction
> > > > > API. If the maintenance is indeed huge, I think it is reasonable to
> > > have
> > > > a
> > > > > special rule for this case at that time.
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > >
> > > > > [1]: https://issues.apache.org/jira/browse/FLINK-28045
> > > > >
> > > > > On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Jingsong,
> > > > > >
> > > > > > Thanks for the reply. I completely agree with you.
> > > > > >
> > > > > > The above 2 options are based on the assumption that the
> community
> > > > cannot
> > > > > > afford to maintain the deprecated DataStream API for long. I'd
> say
> > we
> > > > > > should try everything we can to maintain it for as much time as
> > > > possible.
> > > > > > DataStream API is actually the most used API in Flink by so many
> > > users
> > > > at
> > > > > > this point. Removing it any time soon will dramatically hurt our
> > > users.
> > > > > So
> > > > > > ideally we should keep it for at least 2 years after deprecation,
> > if
> > > > not
> > > > > > more.
> > > > > >
> > > > > > The prohibitively high maintenance overhead is just an
> assumption.
> > > > > > Personally speaking, I don't feel this assumption is necessarily
> > > true.
> > > > We
> > > > > > should re-evaluate once we have the new ProcessFunction API in
> > place.
> > > > > > Without the code it is hard to tell for sure. I am actually kind
> of
> > > > > > optimistic about the maintenance cost.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <
> > jingsonglee0@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Becket and all for your discussion.
> > > > > > >
> > > > > > > > 1. We say this FLIP is enforced starting release 2.0. For
> > current
> > > > 1.x
> > > > > > > APIs,
> > > > > > > we provide a migration period with best effort, while allowing
> > > > > exceptions
> > > > > > > for immediate removal in 2.0. That means we will still try with
> > > best
> > > > > > effort
> > > > > > > to get the ProcessFuncion API ready and deprecate the
> DataStream
> > > API
> > > > in
> > > > > > > 1.x, but will also be allowed to remove DataStream API in 2.0
> if
> > > it's
> > > > > not
> > > > > > > deprecated 2 minor releases before the major version bump.
> > > > > > >
> > > > > > > > 2. We strictly follow the process in this FLIP, and will
> > quickly
> > > > bump
> > > > > > the
> > > > > > > major version from 2.x to 3.0 once the migration period for
> > > > DataStream
> > > > > > API
> > > > > > > is reached.
> > > > > > >
> > > > > > > Sorry, I didn't read the previous detailed discussion because
> the
> > > > > > > discussion list was so long.
> > > > > > >
> > > > > > > I don't really like either of these options.
> > > > > > >
> > > > > > > Considering that DataStream is such an important API, can we
> > offer
> > > a
> > > > > > third
> > > > > > > option:
> > > > > > >
> > > > > > > 3. Maintain the DataStream API throughout 2.X and remove it
> until
> > > > 3.x.
> > > > > > But
> > > > > > > there's no need to assume that 2.X is a short version, it's
> > still a
> > > > > > normal
> > > > > > > major version.
> > > > > > >
> > > > > > > Best,
> > > > > > > Jingsong
> > > > > > >
> > > > > > > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> > > > > > >
> > > > > > > > Thanks much for the input, John, Stefan and Jing.
> > > > > > > >
> > > > > > > > I think Xingtong has well summarized the pros and cons of the
> > two
> > > > > > > options.
> > > > > > > > Let's collect a few more opinions here and we can move
> forward
> > > with
> > > > > the
> > > > > > > one
> > > > > > > > more people prefer.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Thanks Xingtong for the summary. If I could only choose one
> > of
> > > > the
> > > > > > > given
> > > > > > > > > two options, I would go with option 1. I understood that
> > > option 2
> > > > > > > worked
> > > > > > > > > great with Kafka. But the bridge release will still confuse
> > > users
> > > > > and
> > > > > > > my
> > > > > > > > > gut feeling is that many users will skip 2.0 and be waiting
> > for
> > > > 3.0
> > > > > > or
> > > > > > > > even
> > > > > > > > > 3.x. And since fewer users will use Flink 2.x, the
> > development
> > > > > focus
> > > > > > > will
> > > > > > > > > be on Flink 3.0 with the fact that the current Flink
> release
> > is
> > > > > 1.17
> > > > > > > and
> > > > > > > > we
> > > > > > > > > are preparing 2.0 release. That is weird for me.
> > > > > > > > >
> > > > > > > > > THB, I would not name the change from @Public to @Retired
> as
> > a
> > > > > > > demotion.
> > > > > > > > > The purpose of @Retire is to extend the API lifecycle with
> > one
> > > > more
> > > > > > > > stage,
> > > > > > > > > like in the real world, people born, studied, graduated,
> > > worked,
> > > > > and
> > > > > > > > > retired. Afaiu from the previous discussion, there are two
> > > rules
> > > > > we'd
> > > > > > > > like
> > > > > > > > > to follow simultaneously:
> > > > > > > > >
> > > > > > > > > 1. Public APIs can only be changed between major releases.
> > > > > > > > > 2. A smooth migration phase should be offered to users,
> i.e.
> > at
> > > > > > least 2
> > > > > > > > > minor releases after APIs are marked as @deprecated. There
> > > should
> > > > > be
> > > > > > > new
> > > > > > > > > APIs as the replacement.
> > > > > > > > >
> > > > > > > > > Agree, those rules are good to improve the user
> friendliness.
> > > > > Issues
> > > > > > we
> > > > > > > > > discussed are rising because we want to fulfill both of
> them.
> > > If
> > > > we
> > > > > > > take
> > > > > > > > > care of deprecation very seriously, APIs can be marked as
> > > > > > @Deprecated,
> > > > > > > > only
> > > > > > > > > when the new APIs as the replacement provide all
> > > functionalities
> > > > > the
> > > > > > > > > deprecated APIs have. In an ideal case without critical
> bugs
> > > that
> > > > > > might
> > > > > > > > > stop users adopting the new APIs. Otherwise the expected
> > > > > > "replacement"
> > > > > > > > will
> > > > > > > > > not happen. Users will still stick to the deprecated APIs,
> > > > because
> > > > > > the
> > > > > > > > new
> > > > > > > > > APIs can not be used. For big features, it will need at
> > least 4
> > > > > minor
> > > > > > > > > releases(ideal case), i.e. 2+ years to remove deprecated
> > APIs:
> > > > > > > > >
> > > > > > > > > - 1st minor release to build the new APIs as the
> replacement
> > > and
> > > > > > > waiting
> > > > > > > > > for feedback. It might be difficult to mark the old API as
> > > > > deprecated
> > > > > > > in
> > > > > > > > > this release, because we are not sure if the new APIs could
> > > cover
> > > > > > 100%
> > > > > > > > > functionalities.
> > > > > > > > > -  In the lucky case,  mark all old APIs as deprecated in
> the
> > > 2nd
> > > > > > minor
> > > > > > > > > release. (I would even suggest having the new APIs released
> > at
> > > > > least
> > > > > > > for
> > > > > > > > > two minor releases before marking it as deprecated to make
> > sure
> > > > > they
> > > > > > > can
> > > > > > > > > really replace the old APIs, in case we care more about
> > smooth
> > > > > > > migration)
> > > > > > > > > - 3rd minor release for the migration period
> > > > > > > > > -  In another lucky case, the 4th release is a major
> release,
> > > the
> > > > > > > > > deprecated APIs could be removed.
> > > > > > > > >
> > > > > > > > > The above described scenario works only in an ideal case.
> In
> > > > > reality,
> > > > > > > it
> > > > > > > > > might take longer to get the new APIs ready and mark the
> old
> > > API
> > > > > > > > > deprecated. Furthermore, if the 4th release is not a major
> > > > release,
> > > > > > we
> > > > > > > > will
> > > > > > > > > have to maintain both APIs for many further minor releases.
> > The
> > > > > > > question
> > > > > > > > is
> > > > > > > > > how to know the next major release in advance, especially 4
> > > minor
> > > > > > > > releases'
> > > > > > > > > period, i.e. more than 2 years in advance? Given that Flink
> > > > > contains
> > > > > > > many
> > > > > > > > > modules, it is difficult to ask devs to create a 2-3 years
> > > > > > deprecation
> > > > > > > > plan
> > > > > > > > > for each case. In case we want to build major releases at a
> > > fast
> > > > > > pace,
> > > > > > > > > let's say every two years, it means devs must plan any API
> > > > > > deprecation
> > > > > > > > > right after each major release. Afaiac, it is quite
> > difficult.
> > > > > > > > >
> > > > > > > > > The major issue is, afaiu, if we follow rule 2, we have to
> > keep
> > > > all
> > > > > > > > @Public
> > > > > > > > > APIs, e.g. DataStream, that are not marked as deprecated
> yet,
> > > to
> > > > > 2.0.
> > > > > > > > Then
> > > > > > > > > we have to follow rule 1 to keep it unchanged until we have
> > > 3.0.
> > > > > That
> > > > > > > is
> > > > > > > > > why @Retired is useful to give devs more flexibility and
> > still
> > > > > > fulfill
> > > > > > > > both
> > > > > > > > > rules. Let's check it with examples:
> > > > > > > > >
> > > > > > > > > - we have @Public DataStream API in 1.18. It will not be
> > marked
> > > > > > > > > as @Deprecated, because the new APIs as the replacement are
> > not
> > > > > > ready.
> > > > > > > > > - we keep the DataStream API itself unchanged in 2.0, but
> > > change
> > > > > the
> > > > > > > > > annotation from @Public to @Retire. New APIs will be
> > introduced
> > > > > too.
> > > > > > In
> > > > > > > > > this case, Rule 1 is ok, since public API is allowed to
> > change
> > > > > > between
> > > > > > > > > major releases. Only changing annotation is the minimal
> > change
> > > we
> > > > > > could
> > > > > > > > do
> > > > > > > > > and it does not break rule 1. Rule 2 is ok too, since the
> > > > > DataStream
> > > > > > > APIs
> > > > > > > > > work exactly the same.  Attention: the change of @Public ->
> > > > > @Retired
> > > > > > > can
> > > > > > > > > only be done between major releases, because @Public APIs
> can
> > > > only
> > > > > be
> > > > > > > > > changed between major releases.
> > > > > > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > > > > > - in 2.3, DataStream will be removed.
> > > > > > > > >
> > > > > > > > > Becket mentioned previously (please correct me if I didn't
> > > > > understand
> > > > > > > it
> > > > > > > > > correctly) that users might not check the changes of
> > annotation
> > > > and
> > > > > > the
> > > > > > > > > upgrade process might not be smooth. I was wondering, if
> > users
> > > > > don't
> > > > > > > pay
> > > > > > > > > attention to that, they will also not pay attention to
> > > > @deprecated.
> > > > > > The
> > > > > > > > > migration will not be smooth too even when we stick to the
> > Java
> > > > > > > > > standard @deprecated. Let's go throw it with examples:
> > > > > > > > >
> > > > > > > > > Scenario 1: with @Retired
> > > > > > > > >
> > > > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing
> > > should
> > > > > be
> > > > > > > done
> > > > > > > > > for users who still want to use DataStream. The migration
> is
> > > > > smooth.
> > > > > > > But
> > > > > > > > > they should be aware of the annotation changes, i.e.
> @Public
> > ->
> > > > > > > @Retired
> > > > > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is
> > removed.
> > > > > Since
> > > > > > > it
> > > > > > > > is
> > > > > > > > > a major release upgrade, breaking changes are expected.
> > > > > > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is
> > removed.
> > > > > Since
> > > > > > > > APIs
> > > > > > > > > are marked as @Retired, breaking changes are expected too.
> > > > > > > > >
> > > > > > > > > In addition to fulfilling both rules, there are two more
> > > benefits
> > > > > of
> > > > > > > this
> > > > > > > > > option:
> > > > > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case
> 1.
> > > > > > > > > 2. no bridge release is required. DataStream API could be
> > > > > deprecated
> > > > > > in
> > > > > > > > 2.x
> > > > > > > > > without breaking any of the two rules. @Retired provides
> > enough
> > > > > > > > flexibility
> > > > > > > > > for future deprecations and upgrades.
> > > > > > > > >
> > > > > > > > > The only issue I can find is that users might be surprised
> in
> > > > case
> > > > > 3,
> > > > > > > if
> > > > > > > > > they ignore the previous @Public -> @Retired change. But
> for
> > > such
> > > > > > > users,
> > > > > > > > > they will ignore @deprecated annotation too and will have
> > > issues
> > > > > with
> > > > > > > > > pure @deprecated
> > > > > > > > > annotation too:
> > > > > > > > >
> > > > > > > > > Scenario 2: with @deprecated
> > > > > > > > >
> > > > > > > > > The example will be slightly different from scenario 1,
> > because
> > > > > > @Public
> > > > > > > > > APIs can only be removed between major releases. As an
> > example,
> > > > > > > > DataStream
> > > > > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and
> > > then
> > > > > > > removed
> > > > > > > > in
> > > > > > > > > 2.0
> > > > > > > > >
> > > > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not
> > > marked
> > > > > as
> > > > > > > > > deprecated are removed between major releases. Rule 1 is
> ok.
> > > The
> > > > 2
> > > > > > > minor
> > > > > > > > > releases period defined in rule 2 sounds like a smooth
> > > migration
> > > > > > plan,
> > > > > > > > but
> > > > > > > > > for users in this case, it is still not smooth.
> > > > > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade
> > > process
> > > > > > with
> > > > > > > > > deprecated APIs. But for those users who ignore annotation
> > > > changes,
> > > > > > > will
> > > > > > > > > have the same issue mentioned in case 1.
> > > > > > > > >
> > > > > > > > > I understood that, compared to scenario 1, users, who
> ignore
> > > > > @Public
> > > > > > > > > -> @Retired
> > > > > > > > > changes, will not face any breaking changes, after they
> > > migrated
> > > > to
> > > > > > > 2.0.
> > > > > > > > > But the cost of it is bridge releases. There might be many
> > > bridge
> > > > > > > > releases
> > > > > > > > > in the future. As I mentioned previously, it is not easy to
> > > find
> > > > > the
> > > > > > > > right
> > > > > > > > > timing to kick off the deprecation process of APIs before
> the
> > > > next
> > > > > > > > expected
> > > > > > > > > major release and 2-3 years in advance.
> > > > > > > > >
> > > > > > > > > Scenario 3: @PublicEvolving deprecation for users who
> ignore
> > > > > > > @deprecated
> > > > > > > > > annotation changes
> > > > > > > > >
> > > > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20,
> > removed
> > > in
> > > > > > 1.21
> > > > > > > > >
> > > > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change,
> no
> > > > > smooth
> > > > > > > > > migration.
> > > > > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking
> > change,
> > > > > > > > > since @deprecated is ignored
> > > > > > > > >
> > > > > > > > > In summary, for users who care about the annotation change,
> > > they
> > > > > will
> > > > > > > get
> > > > > > > > > more benefits with @Retired. For users who ignore
> annotation
> > > > > changes,
> > > > > > > > they
> > > > > > > > > will be always facing non-smooth upgrades even using pure
> > > > standard
> > > > > > > > > @deprecated
> > > > > > > > > annotation/process.
> > > > > > > > >
> > > > > > > > > For the case that IDEs support @deprecated, that is true.
> > There
> > > > are
> > > > > > no
> > > > > > > > such
> > > > > > > > > supports for homebuilt annotation. But the intention is to
> > let
> > > > > users
> > > > > > be
> > > > > > > > > aware of it. There will be other ways to do it for users
> who
> > > care
> > > > > > about
> > > > > > > > > annotation changes. For users who do not care, there should
> > be
> > > no
> > > > > > > > > difference between with tool support or without. Using
> > @Retired
> > > > > with
> > > > > > > > > @deprecation
> > > > > > > > > together is at least not worse than only using @deprecation
> > > > alone.
> > > > > > And
> > > > > > > > > there are additional benefits with @Retired.
> > > > > > > > >
> > > > > > > > > Just my 2 cents and looking forward to your feedback,
> > > especially
> > > > > > > > different
> > > > > > > > > opinions that will help me understand the issue better.
> > Thanks!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > > > > > <sr...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Xintong,
> > > > > > > > > >
> > > > > > > > > > Thanks for the summary, most of the points that you
> agreed
> > > upon
> > > > > > also
> > > > > > > > make
> > > > > > > > > > sense to me.
> > > > > > > > > >
> > > > > > > > > > >   2. Dropping deprecated Public APIs in minor releases,
> > or
> > > > > > demoting
> > > > > > > > > APIs
> > > > > > > > > > >   from Public to PublicEvolving / Experimental /
> Retired
> > in
> > > > any
> > > > > > > > version
> > > > > > > > > > bump,
> > > > > > > > > > >   are not good practices.
> > > > > > > > > >
> > > > > > > > > > I hope we can can go beyond calling it “not a good
> > practice"
> > > > and
> > > > > > > reach
> > > > > > > > > > consensus that we will not demote or remove public APIs
> in
> > > > minor
> > > > > > > > releases
> > > > > > > > > > and (technically) enforce this rule for all
> contributions.
> > I
> > > > > think
> > > > > > > > it’s a
> > > > > > > > > > reasonable expectation for stability from a project as
> > mature
> > > > as
> > > > > > > Flink.
> > > > > > > > > >
> > > > > > > > > > > I'm personally in favor of option 1. As the
> > > migration-period
> > > > > rule
> > > > > > > is
> > > > > > > > > > newly
> > > > > > > > > > > proposed, I think it's fair to make exceptions for
> cases
> > > > where
> > > > > we
> > > > > > > > > already
> > > > > > > > > > > missed the best chance for planning the deprecation.
> > > > Moreover,
> > > > > I
> > > > > > do
> > > > > > > > > > believe
> > > > > > > > > > > having a quick major version bump does confuse users.
> > Yes,
> > > we
> > > > > can
> > > > > > > > > explain
> > > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> > > anything
> > > > > > other
> > > > > > > > than
> > > > > > > > > > the
> > > > > > > > > > > removal of an deprecated API. But having to explain
> this
> > > > itself
> > > > > > is
> > > > > > > an
> > > > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > > > understanding,
> > > > > > > > his
> > > > > > > > > > > major point is that a quick major version bump causes
> > > barely
> > > > > any
> > > > > > > > actual
> > > > > > > > > > > lose on users, while in option 1 not providing the
> > > migration
> > > > > > period
> > > > > > > > or
> > > > > > > > > > > providing a shorter on is an actual lose on users.
> > > (@Becket,
> > > > > > please
> > > > > > > > > > correct
> > > > > > > > > > > me if I'm mistaken.)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I would be open to both options and maybe postpone this
> > > > decision
> > > > > > > until
> > > > > > > > we
> > > > > > > > > > have the complete roadmap for Flink 2.0. We would have a
> > > better
> > > > > > > > overview
> > > > > > > > > > about the technical consequences, e.g. how hard it would
> be
> > > to
> > > > > > offer
> > > > > > > > and
> > > > > > > > > > maintain both APIs side by side, how well we get both
> APIs
> > > > > > separated,
> > > > > > > > and
> > > > > > > > > > whether or not we will be able to use the full potential
> of
> > > our
> > > > > > > > breaking
> > > > > > > > > > changes before the DataStream API is completely removed.
> > One
> > > > > > example
> > > > > > > > for
> > > > > > > > > > the last point I can think of would be moving from lazy
> to
> > > > eager
> > > > > > > state
> > > > > > > > > > declaration. If we believe that it is reasonable to
> support
> > > all
> > > > > > > planned
> > > > > > > > > > changes while maintaining both APIs, I'm leaning towards
> > > option
> > > > > 2.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stefan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <
> > > > tonysong820@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Becket and I had an offline voice call earlier today.
> We
> > > have
> > > > > > > reached
> > > > > > > > > > > consensus on some of the arguments, but still hold
> > > different
> > > > > > > opinions
> > > > > > > > > on
> > > > > > > > > > > some others. I'd like to post the outcome here for
> > > > > transparency.
> > > > > > > > > > >
> > > > > > > > > > > We both agree that:
> > > > > > > > > > >
> > > > > > > > > > >   1. Providing a migration period would be beneficial
> for
> > > our
> > > > > > > users,
> > > > > > > > > and
> > > > > > > > > > >   we should do that
> > > > > > > > > > >   2. Dropping deprecated Public APIs in minor releases,
> > or
> > > > > > demoting
> > > > > > > > > APIs
> > > > > > > > > > >   from Public to PublicEvolving / Experimental /
> Retired
> > in
> > > > any
> > > > > > > > version
> > > > > > > > > > bump,
> > > > > > > > > > >   are not good practices.
> > > > > > > > > > >   3. Ideally, with this FLIP, developers should be more
> > > > careful
> > > > > > and
> > > > > > > > > plan
> > > > > > > > > > >   API changes ahead. That means:
> > > > > > > > > > >      1. Be more careful with designing APIs and
> promoting
> > > > them
> > > > > to
> > > > > > > > > Public,
> > > > > > > > > > >      so that they won't be changed / removed very soon,
> > and
> > > > > also
> > > > > > > the
> > > > > > > > > > >      maintenance overhead for keeping them after
> > > deprecation
> > > > > > should
> > > > > > > > be
> > > > > > > > > > >      affordable. (I believe this also aligns with
> John's
> > > > > > opinion.)
> > > > > > > > > > >      2. Plan the deprecation / removal of old APIs that
> > we
> > > > > don't
> > > > > > > want
> > > > > > > > > to
> > > > > > > > > > >      carry to the next major version earlier, so that
> > > they'll
> > > > > > have
> > > > > > > > the
> > > > > > > > > > >      2-minor-release migration period before the next
> > major
> > > > > > version
> > > > > > > > > bump.
> > > > > > > > > > >   4. A practical situation is that, we are not ready
> for
> > > > > > > deprecating
> > > > > > > > > the
> > > > > > > > > > >   DataStream API, nor afford to carry it for another
> > couple
> > > > of
> > > > > > > years.
> > > > > > > > > > >      1. We cannot deprecate it now, because the new
> > > > > > ProcessFunction
> > > > > > > > API
> > > > > > > > > > >      (the planned replacement for DataStream API) is
> not
> > > yet
> > > > > > ready.
> > > > > > > > We
> > > > > > > > > > haven't
> > > > > > > > > > >      planned the ProcessFunction API earlier because
> the
> > > > > original
> > > > > > > > > > > plan was to do
> > > > > > > > > > >      in-place changes directly on the DataStream API in
> > > > release
> > > > > > > 2.0,
> > > > > > > > > > until the
> > > > > > > > > > >      smooth migration period is proposed. Before this,
> > the
> > > > only
> > > > > > > > > > guarantees we
> > > > > > > > > > >      provide for Public APIs is that they'll stay
> > > compatible
> > > > > > until
> > > > > > > > the
> > > > > > > > > > next
> > > > > > > > > > >      major version, which led to the understanding the
> > > > in-place
> > > > > > > > changes
> > > > > > > > > > on
> > > > > > > > > > >      Public APIs should be fine.
> > > > > > > > > > >      2. We also cannot afford carrying the DataStream
> API
> > > for
> > > > > > > another
> > > > > > > > > > >      couple of years. One of the reasons that we want
> to
> > > > > > refactor /
> > > > > > > > > > replace
> > > > > > > > > > >      DataStream API is that it exposes and depends on
> too
> > > > many
> > > > > > > > > > > Flink's internal
> > > > > > > > > > >      runtime implementations, which has already
> limited /
> > > > > blocked
> > > > > > > > plans
> > > > > > > > > > for
> > > > > > > > > > >      Flink's internal improvements for many times. Also
> > due
> > > > to
> > > > > > the
> > > > > > > > wide
> > > > > > > > > > and
> > > > > > > > > > >      transitive dependencies on the internals,
> > maintaining
> > > > two
> > > > > > sets
> > > > > > > > of
> > > > > > > > > > lower
> > > > > > > > > > >      APIs itself is expensive and would be better to
> keep
> > > it
> > > > > > short.
> > > > > > > > > This
> > > > > > > > > > is
> > > > > > > > > > >      admittedly a historical design flaw which should
> be
> > > > > avoided
> > > > > > in
> > > > > > > > > > future.
> > > > > > > > > > >
> > > > > > > > > > > What we haven't reached consensus is about how to deal
> > with
> > > > the
> > > > > > > > current
> > > > > > > > > > > situation around the DataStream API. There are two
> > options.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >   1. We say this FLIP is enforced starting release 2.0.
> > For
> > > > > > current
> > > > > > > > 1.x
> > > > > > > > > > >   APIs, we provide a migration period with best effort,
> > > while
> > > > > > > > allowing
> > > > > > > > > > >   exceptions for immediate removal in 2.0. That means
> we
> > > will
> > > > > > still
> > > > > > > > try
> > > > > > > > > > with
> > > > > > > > > > >   best effort to get the ProcessFuncion API ready and
> > > > deprecate
> > > > > > the
> > > > > > > > > > >   DataStream API in 1.x, but will also be allowed to
> > remove
> > > > > > > > DataStream
> > > > > > > > > > API in
> > > > > > > > > > >   2.0 if it's not deprecated 2 minor releases before
> the
> > > > major
> > > > > > > > version
> > > > > > > > > > bump.
> > > > > > > > > > >   2. We strictly follow the process in this FLIP, and
> > will
> > > > > > quickly
> > > > > > > > bump
> > > > > > > > > > >   the major version from 2.x to 3.0 once the migration
> > > period
> > > > > for
> > > > > > > > > > DataStream
> > > > > > > > > > >   API is reached.
> > > > > > > > > > >
> > > > > > > > > > > I'm personally in favor of option 1. As the
> > > migration-period
> > > > > rule
> > > > > > > is
> > > > > > > > > > newly
> > > > > > > > > > > proposed, I think it's fair to make exceptions for
> cases
> > > > where
> > > > > we
> > > > > > > > > already
> > > > > > > > > > > missed the best chance for planning the deprecation.
> > > > Moreover,
> > > > > I
> > > > > > do
> > > > > > > > > > believe
> > > > > > > > > > > having a quick major version bump does confuse users.
> > Yes,
> > > we
> > > > > can
> > > > > > > > > explain
> > > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> > > anything
> > > > > > other
> > > > > > > > than
> > > > > > > > > > the
> > > > > > > > > > > removal of an deprecated API. But having to explain
> this
> > > > itself
> > > > > > is
> > > > > > > an
> > > > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > > > understanding,
> > > > > > > > his
> > > > > > > > > > > major point is that a quick major version bump causes
> > > barely
> > > > > any
> > > > > > > > actual
> > > > > > > > > > > lose on users, while in option 1 not providing the
> > > migration
> > > > > > period
> > > > > > > > or
> > > > > > > > > > > providing a shorter on is an actual lose on users.
> > > (@Becket,
> > > > > > please
> > > > > > > > > > correct
> > > > > > > > > > > me if I'm mistaken.)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > And we'd like to hear more feedback from the community.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Xintong
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > > > > > vvcephei@apache.org
> > > > > > > > > > <ma...@apache.org>> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi Becket and Xintong,
> > > > > > > > > > >>
> > > > > > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > > > > > >>
> > > > > > > > > > >> Once an API is marked Public, we're committing to
> > support
> > > it
> > > > > > until
> > > > > > > > > it's
> > > > > > > > > > >> deprecated, and once it's deprecated, to leave it in
> > place
> > > > for
> > > > > > at
> > > > > > > > > least
> > > > > > > > > > two
> > > > > > > > > > >> minor releases and then only remove it in the
> following
> > > > major
> > > > > > > > release.
> > > > > > > > > > >>
> > > > > > > > > > >> As a user, I would be dismayed to discover that the
> > > project
> > > > > > > > > maintainers
> > > > > > > > > > >> demoted a Public API to PublicEvolving or Experimental
> > in
> > > > > order
> > > > > > to
> > > > > > > > > > violate
> > > > > > > > > > >> the spirit of the deprecation period for Public APIs.
> > > > > Likewise,
> > > > > > > I'd
> > > > > > > > > > view a
> > > > > > > > > > >> rapid sequence of minor releases for the purpose of
> > > dropping
> > > > > > > > > deprecated
> > > > > > > > > > >> APIs in the next major release as another violation of
> > the
> > > > > > spirit
> > > > > > > of
> > > > > > > > > our
> > > > > > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > > > > > >>
> > > > > > > > > > >> I'm glad we're gaming out these situations before we
> > > > institute
> > > > > > > this
> > > > > > > > > > FLIP,
> > > > > > > > > > >> and I hope we all converge on a clear understanding of
> > > what
> > > > > > we're
> > > > > > > > > > >> guaranteeing. From where I'm sitting, the essence of
> > this
> > > > FLIP
> > > > > > is
> > > > > > > > > > >> specifically to prevent us from doing whatever we want
> > and
> > > > > > instead
> > > > > > > > to
> > > > > > > > > > >> commit to providing a suitable notice period to users
> > who
> > > > rely
> > > > > > on
> > > > > > > > > Public
> > > > > > > > > > >> APIs.
> > > > > > > > > > >>
> > > > > > > > > > >> It honestly shouldn't matter what the maintenance
> > overhead
> > > > is
> > > > > > (or,
> > > > > > > > > > rather,
> > > > > > > > > > >> evaluating the maintenance overhead should be a
> > > > consideration
> > > > > > > before
> > > > > > > > > we
> > > > > > > > > > >> promote an API to Public to begin with). My experience
> > in
> > > > > other
> > > > > > > > > > projects is
> > > > > > > > > > >> that regretting a feature so much you want to drop it
> > asap
> > > > is
> > > > > > > > > extremely
> > > > > > > > > > >> rare, and that living with the pain for a little while
> > > until
> > > > > the
> > > > > > > > > > >> deprecation period expires will probably make us
> better
> > > > > > engineers.
> > > > > > > > > > >>
> > > > > > > > > > >> With that all said, one thing that is not a violation
> is
> > > to
> > > > > > > propose
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > >> major release in order to drop especially burdensome
> > > > > deprecated
> > > > > > > APIs
> > > > > > > > > > that
> > > > > > > > > > >> have already enjoyed their full deprecation period.
> > Those
> > > > > really
> > > > > > > bad
> > > > > > > > > > "case
> > > > > > > > > > >> A" features will be very rare.
> > > > > > > > > > >>
> > > > > > > > > > >> Looking over the FLIP again, I see a reference to the
> > > > release
> > > > > > > > > > >> documentation, but not to the build tooling support
> for
> > > the
> > > > > > > > Deprecated
> > > > > > > > > > >> annotation, so it might help to share this scenario:
> The
> > > way
> > > > > > this
> > > > > > > > will
> > > > > > > > > > all
> > > > > > > > > > >> come together in practice is that, by giving good
> > > stability
> > > > > > > > > guarantees,
> > > > > > > > > > we
> > > > > > > > > > >> will encourage our users to bump minor versions when
> > they
> > > > are
> > > > > > > > > released,
> > > > > > > > > > >> which means that they will see compiler warnings
> > > immediately
> > > > > if
> > > > > > > any
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> APIs they've used are deprecated. In fact, I'd
> recommend
> > > for
> > > > > > them
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > >> the `-Werror` javac flag to be sure that they don't
> miss
> > > > those
> > > > > > > > > warnings.
> > > > > > > > > > >> Given our release cadence, this means that they will
> > have
> > > > > about
> > > > > > > six
> > > > > > > > > > months
> > > > > > > > > > >> to migrate off of any deprecated APIs without being
> > forced
> > > > to
> > > > > > > avoid
> > > > > > > > > > >> updates. And they really need to be able to adopt
> > further
> > > > > minor
> > > > > > > > > updates
> > > > > > > > > > in
> > > > > > > > > > >> the series, since they may contain further
> improvements
> > > that
> > > > > > > > actually
> > > > > > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > > > > > >>
> > > > > > > > > > >> The compiler warnings are a far more reliable
> mechanism
> > to
> > > > > > > advertise
> > > > > > > > > > >> deprecation than release notes, since users' own
> builds
> > > will
> > > > > > > notify
> > > > > > > > > them
> > > > > > > > > > >> right away about only the deprecations that are
> relevant
> > > to
> > > > > > them,
> > > > > > > > but
> > > > > > > > > it
> > > > > > > > > > >> does rely on users feeling ok to bump up minor
> versions
> > > in a
> > > > > > > timely
> > > > > > > > > > >> fashion. If we consistently burn them by not observing
> > the
> > > > > > > stability
> > > > > > > > > > >> guarantees, they'll try to avoid updates instead, and
> > the
> > > > > whole
> > > > > > > > thing
> > > > > > > > > > falls
> > > > > > > > > > >> apart.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks again,
> > > > > > > > > > >> -John
> > > > > > > > > > >>
> > > > > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > > > > > >>> Hi Xintong,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Let's compare the following cases:
> > > > > > > > > > >>>
> > > > > > > > > > >>> A. If the maintenance overhead of the deprecated API
> is
> > > > high,
> > > > > > and
> > > > > > > > we
> > > > > > > > > > want
> > > > > > > > > > >>> to remove it after two minor releases. Then there are
> > two
> > > > > > > options:
> > > > > > > > > > >>>    A1: Demote the API in the major version bump and
> > > remove
> > > > > the
> > > > > > > code
> > > > > > > > > > >> with a
> > > > > > > > > > >>> minor version bump.
> > > > > > > > > > >>>    A2: Do exactly the same as 1-A, except that when
> > > > removing
> > > > > > the
> > > > > > > > > code,
> > > > > > > > > > >>> bump up the major version. So this might be a short
> > major
> > > > > > > version.
> > > > > > > > > > >>>
> > > > > > > > > > >>> B. If the maintenance overhead of the deprecated API
> is
> > > not
> > > > > > high,
> > > > > > > > > > >> therefore
> > > > > > > > > > >>> keeping it for a long time is affordable. There are
> > also
> > > > two
> > > > > > > > options:
> > > > > > > > > > >>>    B1: Same as A-1, demote the API in the major
> version
> > > > bump
> > > > > > and
> > > > > > > > > remove
> > > > > > > > > > >>> the code with a minor version bump.
> > > > > > > > > > >>>    B2: Keep the API for all the minor versions in the
> > > major
> > > > > > > > version,
> > > > > > > > > > and
> > > > > > > > > > >>> only remove the code in the next major version.
> > > > > > > > > > >>>
> > > > > > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > > > > > >>>
> > > > > > > > > > >>> For case A:
> > > > > > > > > > >>> The following stays the same for A1 and A2:
> > > > > > > > > > >>>  - users will lose the API after two minor leases. So
> > the
> > > > > > > migration
> > > > > > > > > > >> period
> > > > > > > > > > >>> is the same for A-1 and A-2.
> > > > > > > > > > >>>  - A1 and A2 will have exactly the same code after
> the
> > > > > removal
> > > > > > of
> > > > > > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > > > > > >>>  - To move forward, users need to move to the next
> > minor
> > > > > > version
> > > > > > > in
> > > > > > > > > A1,
> > > > > > > > > > >> or
> > > > > > > > > > >>> to the next major version in A2. Because the code are
> > the
> > > > > same,
> > > > > > > the
> > > > > > > > > > >> actual
> > > > > > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > > > > > >>>
> > > > > > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > > > > > >>>  - A1 allows keeping the major version release
> cadence.
> > > A-2
> > > > > > will
> > > > > > > > > have a
> > > > > > > > > > >>> short major version release.
> > > > > > > > > > >>>  - A1 breaks the well understood API semantic, while
> > A-2
> > > > does
> > > > > > > not.
> > > > > > > > > > >>>
> > > > > > > > > > >>> From what I see, since there is no well establish
> > > standard
> > > > > > > > regarding
> > > > > > > > > > how
> > > > > > > > > > >>> long a major version should be released, A short
> major
> > > > > version
> > > > > > > > > release
> > > > > > > > > > is
> > > > > > > > > > >>> potential and emotional. It is not ideal but does not
> > > have
> > > > > > > material
> > > > > > > > > > >>> downsides compared with A1. I did not hear anyone
> > > > complaining
> > > > > > > about
> > > > > > > > > > Kafka
> > > > > > > > > > >>> only has two 1.x release. However, A1 actually breaks
> > the
> > > > > well
> > > > > > > > > > understood
> > > > > > > > > > >>> API semantic, which has more material impact.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs
> should
> > > > fall
> > > > > > into
> > > > > > > > > case
> > > > > > > > > > B.
> > > > > > > > > > >>> So, short major versions should be very occasional.
> I'd
> > > be
> > > > > very
> > > > > > > > > > concerned
> > > > > > > > > > >>> if the reason we choose A1 is simply because we
> cannot
> > > > afford
> > > > > > > > > > >> maintaining a
> > > > > > > > > > >>> bunch of deprecated APIs until the next major
> version.
> > > This
> > > > > > > > indicates
> > > > > > > > > > >> that
> > > > > > > > > > >>> the actual problem we need to solve is to lower the
> > > > > maintenance
> > > > > > > > > > overhead
> > > > > > > > > > >> of
> > > > > > > > > > >>> deprecated APIs, so that we are comfortable to keep
> > them
> > > > > > longer.
> > > > > > > As
> > > > > > > > > > John
> > > > > > > > > > >>> and I mentioned earlier, there are ways to achieve
> this
> > > and
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > >>> learn how to do it in Flink. Otherwise, our
> discussion
> > > > about
> > > > > > > > > versioning
> > > > > > > > > > >>> here does not bring much value, because we will end
> up
> > > > with a
> > > > > > > bunch
> > > > > > > > > of
> > > > > > > > > > >>> short-lived APIs which upset our users, no matter how
> > we
> > > > > > version
> > > > > > > > the
> > > > > > > > > > >>> releases.
> > > > > > > > > > >>>
> > > > > > > > > > >>> So, if there are concrete examples that you think
> will
> > > > block
> > > > > us
> > > > > > > > from
> > > > > > > > > > >>> keeping API stability with affordable cost, let's
> take
> > a
> > > > look
> > > > > > > > > together
> > > > > > > > > > >> and
> > > > > > > > > > >>> see if that can be improved.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jiangjie (Becket) Qin
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> From what I see,
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > > > > > tonysong820@gmail.com
> > > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> The part I don't understand is if we are willing to
> > > have
> > > > a
> > > > > > > > > migration
> > > > > > > > > > >>>>> period, and do a minor version bump to remove an
> API,
> > > > what
> > > > > do
> > > > > > > we
> > > > > > > > > > >> lose to
> > > > > > > > > > >>>> do
> > > > > > > > > > >>>>> a major version bump instead, so we don't break the
> > > > common
> > > > > > > > > versioning
> > > > > > > > > > >>>>> semantic?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think we are talking about the cases where a major
> > > > version
> > > > > > > bump
> > > > > > > > > > >> happens
> > > > > > > > > > >>>> before a deprecated Public API reaches its migration
> > > > period.
> > > > > > So
> > > > > > > > > > >> removing
> > > > > > > > > > >>>> the API with another major version bump means we
> have
> > > two
> > > > > > > > > consecutive
> > > > > > > > > > >> major
> > > > > > > > > > >>>> versions in a very short time. And as previously
> > > > mentioned,
> > > > > > > having
> > > > > > > > > > >> frequent
> > > > > > > > > > >>>> major version bumps would weaken the value of the
> > > > commitment
> > > > > > > > "Public
> > > > > > > > > > >> API
> > > > > > > > > > >>>> stay compatible within a major version". I think
> users
> > > > also
> > > > > > have
> > > > > > > > > > >>>> expectations about how long a major version should
> > live,
> > > > > which
> > > > > > > > > should
> > > > > > > > > > >> be at
> > > > > > > > > > >>>> least a couple of years rather than 1-2 minor
> > releases.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> This is another option, but I think it is very
> likely
> > > more
> > > > > > > > expensive
> > > > > > > > > > >> than
> > > > > > > > > > >>>>> simply bumping the major version. And I imagine
> there
> > > > will
> > > > > be
> > > > > > > > > > >> questions
> > > > > > > > > > >>>>> like "why this feature is in 1.20, but does not
> exist
> > > in
> > > > > > 2.0"?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Yes, it's more expensive than simply bumping the
> major
> > > > > > version,
> > > > > > > > but
> > > > > > > > > > >> IMHO
> > > > > > > > > > >>>> it's probably cheaper than carrying the API until
> the
> > > next
> > > > > > major
> > > > > > > > > > >> version if
> > > > > > > > > > >>>> we don't bump the major version very soon. And see
> my
> > > > reply
> > > > > > > above
> > > > > > > > on
> > > > > > > > > > >> why
> > > > > > > > > > >>>> not bumping immediately.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Best,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Xintong
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > > > > > becket.qin@gmail.com
> > > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Hi Xintong,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Please see the replies below.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I see your point, that users would feel surprised
> if
> > > they
> > > > > > find
> > > > > > > > > > >> things no
> > > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> > > release.
> > > > > > > > However,
> > > > > > > > > > >> I'd
> > > > > > > > > > >>>>> like
> > > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have
> the
> > > > > similar
> > > > > > > > > > >> problem
> > > > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > > > attention
> > > > > > > and
> > > > > > > > > > >> make
> > > > > > > > > > >>>>> sure
> > > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> > > longer
> > > > > be
> > > > > > > > Public
> > > > > > > > > > >> in
> > > > > > > > > > >>>>> 2.0.
> > > > > > > > > > >>>>>> There're various ways to do that, e.g., release
> > notes,
> > > > > > > warnings
> > > > > > > > in
> > > > > > > > > > >>>> logs,
> > > > > > > > > > >>>>>> etc.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> First of all, I am not a fan of removing
> > PublicEvolving
> > > > > APIs
> > > > > > in
> > > > > > > > > minor
> > > > > > > > > > >>>>> version changes. Personally speaking, I want them
> to
> > be
> > > > > also
> > > > > > > > > removed
> > > > > > > > > > >> in
> > > > > > > > > > >>>>> major version changes.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Empirically, I rarely see projects with complex API
> > > > > semantic
> > > > > > > work
> > > > > > > > > > >> well.
> > > > > > > > > > >>>>> Many, if not most, users don't read docs / release
> > > notes
> > > > /
> > > > > > > > warnings
> > > > > > > > > > >> in
> > > > > > > > > > >>>>> logs. I don't think that is their fault. Typically
> an
> > > > > > > application
> > > > > > > > > > >>>> developer
> > > > > > > > > > >>>>> will have to deal with dozens of libraries
> maintained
> > > by
> > > > > > > various
> > > > > > > > > > >>>>> communities / groups. It is just too difficult for
> > them
> > > > to
> > > > > > keep
> > > > > > > > > > >> track of
> > > > > > > > > > >>>>> all the specific semantics each project puts there.
> > If
> > > > you
> > > > > > > think
> > > > > > > > > > >> about
> > > > > > > > > > >>>> it,
> > > > > > > > > > >>>>> how many of Flink developers read all the release
> > notes
> > > > of
> > > > > > > Guava,
> > > > > > > > > > >> Apache
> > > > > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you
> > upgrade a
> > > > > > version
> > > > > > > > of
> > > > > > > > > > >> them?
> > > > > > > > > > >>>>> One would probably try bumping the dependency
> version
> > > and
> > > > > see
> > > > > > > if
> > > > > > > > > > >> there is
> > > > > > > > > > >>>>> an exception and solve them case by case. And if it
> > is
> > > a
> > > > > > minor
> > > > > > > > > > >> version
> > > > > > > > > > >>>>> bump, one probably does not expect an exception at
> > all.
> > > > > > Another
> > > > > > > > > > >> example
> > > > > > > > > > >>>> is
> > > > > > > > > > >>>>> that in Flink we still have so many usage of
> > deprecated
> > > > > > methods
> > > > > > > > all
> > > > > > > > > > >> over
> > > > > > > > > > >>>>> the place, and I strongly doubt everyone knows when
> > the
> > > > > > source
> > > > > > > > code
> > > > > > > > > > >> of
> > > > > > > > > > >>>>> these methods are deprecated and when they should
> be
> > > > > removed.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> So, the most important principle of API is simple
> and
> > > > > > > intuitive.
> > > > > > > > > The
> > > > > > > > > > >>>>> versioning semantic is a simple and universally
> > > accepted
> > > > > API
> > > > > > > > > > >> stability
> > > > > > > > > > >>>>> standard. If we ourselves as Flink developers are
> > > relying
> > > > > on
> > > > > > > this
> > > > > > > > > > >> for our
> > > > > > > > > > >>>>> own dependencies. I don't think we can expect more
> > from
> > > > our
> > > > > > > > users.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Another possible alternative: whenever there's a
> > > > deprecated
> > > > > > > > Public
> > > > > > > > > > >> API
> > > > > > > > > > >>>> that
> > > > > > > > > > >>>>>> reaches a major version bump before the migration
> > > > period,
> > > > > > and
> > > > > > > we
> > > > > > > > > > >> also
> > > > > > > > > > >>>>> don't
> > > > > > > > > > >>>>>> want to carry it for all the next major release
> > > series,
> > > > we
> > > > > > may
> > > > > > > > > > >> consider
> > > > > > > > > > >>>>>> releasing more minor releases for the previous
> major
> > > > > version
> > > > > > > > after
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19,
> and
> > > > then
> > > > > we
> > > > > > > > bump
> > > > > > > > > > >> to
> > > > > > > > > > >>>> 2.0,
> > > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That
> should
> > > > > provide
> > > > > > > > users
> > > > > > > > > > >>>> another
> > > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while
> > satisfying
> > > > the
> > > > > > > > > > >>>> 2-minor-release
> > > > > > > > > > >>>>>> migration period.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> This is another option, but I think it is very
> likely
> > > > more
> > > > > > > > > expensive
> > > > > > > > > > >> than
> > > > > > > > > > >>>>> simply bumping the major version. And I imagine
> there
> > > > will
> > > > > be
> > > > > > > > > > >> questions
> > > > > > > > > > >>>>> like "why this feature is in 1.20, but does not
> exist
> > > in
> > > > > > 2.0"?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I think my major point is, we should not carry APIs
> > > > > > deprecated
> > > > > > > > in a
> > > > > > > > > > >>>>>> previous major version along all the next major
> > > version
> > > > > > > series.
> > > > > > > > > I'd
> > > > > > > > > > >>>> like
> > > > > > > > > > >>>>> to
> > > > > > > > > > >>>>>> try giving users more commitments, i.e. the
> > migration
> > > > > > period,
> > > > > > > as
> > > > > > > > > > >> long
> > > > > > > > > > >>>> as
> > > > > > > > > > >>>>> it
> > > > > > > > > > >>>>>> does not prevent us from making breaking changes.
> If
> > > it
> > > > > > > doesn't
> > > > > > > > > > >> work,
> > > > > > > > > > >>>> I'd
> > > > > > > > > > >>>>>> be in favor of not providing the migration period,
> > but
> > > > > > > fallback
> > > > > > > > to
> > > > > > > > > > >> only
> > > > > > > > > > >>>>>> guarantee the compatibility within the major
> > version.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> The part I don't understand is if we are willing to
> > > have
> > > > a
> > > > > > > > > migration
> > > > > > > > > > >>>>> period, and do a minor version bump to remove an
> API,
> > > > what
> > > > > do
> > > > > > > we
> > > > > > > > > > >> lose to
> > > > > > > > > > >>>> do
> > > > > > > > > > >>>>> a major version bump instead, so we don't break the
> > > > common
> > > > > > > > > versioning
> > > > > > > > > > >>>>> semantic?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thanks,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > > > > > tonysong820@gmail.com
> > > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > > >>>>> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> As an end user who only uses Public APIs, if I
> > don't
> > > > > change
> > > > > > > my
> > > > > > > > > > >> code
> > > > > > > > > > >>>> at
> > > > > > > > > > >>>>>>> all, my expectation is the following:
> > > > > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > > issue, I
> > > > > am
> > > > > > > fine
> > > > > > > > > > >> with
> > > > > > > > > > >>>>> all
> > > > > > > > > > >>>>>>> the 2.x versions.
> > > > > > > > > > >>>>>>> Actually I think there are some dependency
> version
> > > > > > resolution
> > > > > > > > > > >>>> policies
> > > > > > > > > > >>>>>> out
> > > > > > > > > > >>>>>>> there which picks the highest minor version when
> > the
> > > > > > > > dependencies
> > > > > > > > > > >>>> pull
> > > > > > > > > > >>>>> in
> > > > > > > > > > >>>>>>> multiple minor versions of the same jar, which
> may
> > be
> > > > > > broken
> > > > > > > if
> > > > > > > > > > >> we
> > > > > > > > > > >>>>> remove
> > > > > > > > > > >>>>>>> the API in minor releases.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> I see your point, that users would feel surprised
> if
> > > > they
> > > > > > find
> > > > > > > > > > >> things
> > > > > > > > > > >>>> no
> > > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> > > release.
> > > > > > > > However,
> > > > > > > > > > >> I'd
> > > > > > > > > > >>>>> like
> > > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have
> the
> > > > > similar
> > > > > > > > > > >> problem
> > > > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > > > attention
> > > > > > > and
> > > > > > > > > > >> make
> > > > > > > > > > >>>>> sure
> > > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> > > longer
> > > > > be
> > > > > > > > Public
> > > > > > > > > > >> in
> > > > > > > > > > >>>>> 2.0.
> > > > > > > > > > >>>>>> There're various ways to do that, e.g., release
> > notes,
> > > > > > > warnings
> > > > > > > > in
> > > > > > > > > > >>>> logs,
> > > > > > > > > > >>>>>> etc.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Another possible alternative: whenever there's a
> > > > > deprecated
> > > > > > > > Public
> > > > > > > > > > >> API
> > > > > > > > > > >>>>> that
> > > > > > > > > > >>>>>> reaches a major version bump before the migration
> > > > period,
> > > > > > and
> > > > > > > we
> > > > > > > > > > >> also
> > > > > > > > > > >>>>> don't
> > > > > > > > > > >>>>>> want to carry it for all the next major release
> > > series,
> > > > we
> > > > > > may
> > > > > > > > > > >> consider
> > > > > > > > > > >>>>>> releasing more minor releases for the previous
> major
> > > > > version
> > > > > > > > after
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19,
> and
> > > > then
> > > > > we
> > > > > > > > bump
> > > > > > > > > > >> to
> > > > > > > > > > >>>> 2.0,
> > > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That
> should
> > > > > provide
> > > > > > > > users
> > > > > > > > > > >>>> another
> > > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while
> > satisfying
> > > > the
> > > > > > > > > > >>>> 2-minor-release
> > > > > > > > > > >>>>>> migration period.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> I think my major point is, we should not carry
> APIs
> > > > > > deprecated
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > >>>>>> previous major version along all the next major
> > > version
> > > > > > > series.
> > > > > > > > > I'd
> > > > > > > > > > >>>> like
> > > > > > > > > > >>>>> to
> > > > > > > > > > >>>>>> try giving users more commitments, i.e. the
> > migration
> > > > > > period,
> > > > > > > as
> > > > > > > > > > >> long
> > > > > > > > > > >>>> as
> > > > > > > > > > >>>>> it
> > > > > > > > > > >>>>>> does not prevent us from making breaking changes.
> If
> > > it
> > > > > > > doesn't
> > > > > > > > > > >> work,
> > > > > > > > > > >>>> I'd
> > > > > > > > > > >>>>>> be in favor of not providing the migration period,
> > but
> > > > > > > fallback
> > > > > > > > to
> > > > > > > > > > >> only
> > > > > > > > > > >>>>>> guarantee the compatibility within the major
> > version.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Best,
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Xintong
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > > > > > vvcephei@apache.org
> > > > > > > > > > <ma...@apache.org>
> > > > > > > > > > >>>
> > > > > > > > > > >>>>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Hi Becket,
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> > > > > conversation
> > > > > > > > about
> > > > > > > > > > >>>>>>> compatibility outside of this FLIP thread, but
> for
> > > > now, I
> > > > > > can
> > > > > > > > > > >> accept
> > > > > > > > > > >>>>> your
> > > > > > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Thanks again,
> > > > > > > > > > >>>>>>> John
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > > > > >>>>>>>> Hi John,
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in
> > > major
> > > > > > > > releases
> > > > > > > > > > >>>>> across
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > > > PublicEvolving
> > > > > > > APIs
> > > > > > > > > > >> are
> > > > > > > > > > >>>> by
> > > > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> > > should
> > > > > be
> > > > > > > > > > >> reflected
> > > > > > > > > > >>>>> in
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> > > must
> > > > > keep
> > > > > > > > them
> > > > > > > > > > >>>>> around
> > > > > > > > > > >>>>>>> for
> > > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we
> > can
> > > > > drop
> > > > > > > them
> > > > > > > > > > >> in a
> > > > > > > > > > >>>>>>> minor or
> > > > > > > > > > >>>>>>>>> patch release.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Personally speaking, I would love to do this,
> for
> > > > > exactly
> > > > > > > the
> > > > > > > > > > >>>> reason
> > > > > > > > > > >>>>>> you
> > > > > > > > > > >>>>>>>> mentioned. However, I did not propose this due
> to
> > > the
> > > > > > > > following
> > > > > > > > > > >>>>>> reasons:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing
> the
> > > > > > accepted
> > > > > > > > > > >> FLIPs
> > > > > > > > > > >>>> too
> > > > > > > > > > >>>>>>> soon.
> > > > > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > > > > > development.
> > > > > > > At
> > > > > > > > > > >> this
> > > > > > > > > > >>>>>>> point,
> > > > > > > > > > >>>>>>>> Flink still lacks some design / routines to
> > support
> > > > good
> > > > > > API
> > > > > > > > > > >>>>>>> evolvability /
> > > > > > > > > > >>>>>>>> extensibility. Just like you said, it takes some
> > > time
> > > > to
> > > > > > be
> > > > > > > > > > >> good at
> > > > > > > > > > >>>>>> this.
> > > > > > > > > > >>>>>>>> In this case, my concern is that only removing
> > > > > > Experimental
> > > > > > > /
> > > > > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may
> > > > result
> > > > > in
> > > > > > > too
> > > > > > > > > > >> much
> > > > > > > > > > >>>>>>>> overhead and dramatically slow down the
> > development
> > > of
> > > > > > > Flink.
> > > > > > > > > > >> So, I
> > > > > > > > > > >>>>> was
> > > > > > > > > > >>>>>>>> thinking that we can start with the current
> > status.
> > > > > > > Hopefully
> > > > > > > > > > >> after
> > > > > > > > > > >>>>> we
> > > > > > > > > > >>>>>>> are
> > > > > > > > > > >>>>>>>> more comfortable with the maintenance overhead
> of
> > > > > > deprecated
> > > > > > > > > > >> APIs,
> > > > > > > > > > >>>> we
> > > > > > > > > > >>>>>> can
> > > > > > > > > > >>>>>>>> then have a stronger guarantee for Experimental
> /
> > > > > > > > > > >> PublicEvolving
> > > > > > > > > > >>>>> APIs.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Hi Becket,
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation
> > process
> > > is
> > > > > > > really
> > > > > > > > > > >>>>>> important.
> > > > > > > > > > >>>>>>> I
> > > > > > > > > > >>>>>>>>> understand some people’s concerns about the
> > > > additional
> > > > > > > burden
> > > > > > > > > > >> for
> > > > > > > > > > >>>>>>> project
> > > > > > > > > > >>>>>>>>> maintainers, but my personal experience with
> > Kafka
> > > > has
> > > > > > been
> > > > > > > > > > >> that
> > > > > > > > > > >>>>> it’s
> > > > > > > > > > >>>>>>> very
> > > > > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit
> to
> > > > users.
> > > > > > In
> > > > > > > > > > >> fact,
> > > > > > > > > > >>>>> users
> > > > > > > > > > >>>>>>>>> being able to confidently upgrade is also a
> > benefit
> > > > to
> > > > > > > > > > >>>> maintainers,
> > > > > > > > > > >>>>> as
> > > > > > > > > > >>>>>>> we
> > > > > > > > > > >>>>>>>>> will get fewer questions from people stuck on
> > very
> > > > old
> > > > > > > > > > >> versions.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> One question:
> > > > > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs
> in
> > > > major
> > > > > > > > > > >> releases
> > > > > > > > > > >>>>> across
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > > > PublicEvolving
> > > > > > > APIs
> > > > > > > > > > >> are
> > > > > > > > > > >>>> by
> > > > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> > > should
> > > > > be
> > > > > > > > > > >> reflected
> > > > > > > > > > >>>>> in
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> > > must
> > > > > keep
> > > > > > > > them
> > > > > > > > > > >>>>> around
> > > > > > > > > > >>>>>>> for
> > > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we
> > can
> > > > > drop
> > > > > > > them
> > > > > > > > > > >> in a
> > > > > > > > > > >>>>>>> minor or
> > > > > > > > > > >>>>>>>>> patch release.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> The advantage of forbidding the removal of any
> > API
> > > in
> > > > > > minor
> > > > > > > > or
> > > > > > > > > > >>>> patch
> > > > > > > > > > >>>>>>>>> releases is that users will get a strong
> > guarantee
> > > > that
> > > > > > > they
> > > > > > > > > > >> can
> > > > > > > > > > >>>>> bump
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>> minor or patch version and still be able to
> > > compile,
> > > > or
> > > > > > > even
> > > > > > > > > > >> just
> > > > > > > > > > >>>>>>> re-link
> > > > > > > > > > >>>>>>>>> and know that they won’t face “MethodDef”
> > > exceptions
> > > > at
> > > > > > run
> > > > > > > > > > >> time.
> > > > > > > > > > >>>>> This
> > > > > > > > > > >>>>>>> is a
> > > > > > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> > > > > Experimental
> > > > > > > > APIs
> > > > > > > > > > >>>>> outside
> > > > > > > > > > >>>>>>> of
> > > > > > > > > > >>>>>>>>> major releases, users can no longer confidently
> > > > > upgrade.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a
> couple
> > > of
> > > > > > > points:
> > > > > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation
> > > instead
> > > > of
> > > > > > > > > > >>>> introducing
> > > > > > > > > > >>>>>> our
> > > > > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is
> > well
> > > > > > > > integrated
> > > > > > > > > > >>>> into
> > > > > > > > > > >>>>>>> build
> > > > > > > > > > >>>>>>>>> tools and IDEs.
> > > > > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in
> > this
> > > > > FLIP;
> > > > > > > it
> > > > > > > > > > >> seems
> > > > > > > > > > >>>>>>>>> orthogonal, and something that should probably
> be
> > > > taken
> > > > > > > > > > >>>> case-by-case
> > > > > > > > > > >>>>>>>>> anyway.
> > > > > > > > > > >>>>>>>>> * Aside from deprecation and removal, there
> have
> > > been
> > > > > > some
> > > > > > > > > > >>>>> discussions
> > > > > > > > > > >>>>>>>>> about how to evolve APIs and behavior in
> > compatible
> > > > > ways.
> > > > > > > > > > >> This is
> > > > > > > > > > >>>>>>> somewhat
> > > > > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with
> it
> > > > > before,
> > > > > > > > it’ll
> > > > > > > > > > >>>> take
> > > > > > > > > > >>>>>> some
> > > > > > > > > > >>>>>>>>> time to become good at it. I feel like this
> topic
> > > > > should
> > > > > > > also
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>>> orthogonal
> > > > > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be
> to
> > > > > adopt a
> > > > > > > > > > >> simple
> > > > > > > > > > >>>>>> policy
> > > > > > > > > > >>>>>>> not
> > > > > > > > > > >>>>>>>>> to break existing user programs, and leave the
> > > “how”
> > > > up
> > > > > > to
> > > > > > > > > > >>>>>> implementers
> > > > > > > > > > >>>>>>> and
> > > > > > > > > > >>>>>>>>> reviewers.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > > >>>>>>>>> John
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > > > > >>>>>>>>>> Hi All,
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by
> > Xintong
> > > > is
> > > > > a
> > > > > > > > > > >> great
> > > > > > > > > > >>>>> idea.
> > > > > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired,
> > i.e.
> > > > > > > > > > >>>> @PublicEvolving
> > > > > > > > > > >>>>>> --(2
> > > > > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1
> > > major
> > > > > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a
> > lot
> > > of
> > > > > > > > > > >> flexibility
> > > > > > > > > > >>>>>>> without
> > > > > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are
> > > allowed
> > > > to
> > > > > > > > > > >> change
> > > > > > > > > > >>>>>> between
> > > > > > > > > > >>>>>>>>> major
> > > > > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable
> and
> > > > > > provides
> > > > > > > > > > >>>>> additional
> > > > > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the
> APIs
> > > > > > themself
> > > > > > > > > > >> are
> > > > > > > > > > >>>> not
> > > > > > > > > > >>>>>>>>> changed.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> > > > > issues. I
> > > > > > > > > > >> want to
> > > > > > > > > > >>>>>> move
> > > > > > > > > > >>>>>>> one
> > > > > > > > > > >>>>>>>>>> step further and suggest introducing one more
> > > > > annotation
> > > > > > > > > > >>>> @Retired.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a
> > compromise
> > > of
> > > > > > > > > > >> downgrading
> > > > > > > > > > >>>>>>> @Public
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my
> > > reply,
> > > > > > Java
> > > > > > > > > > >>>> standard
> > > > > > > > > > >>>>>>>>>> @deprecated should be used in the early stage
> of
> > > the
> > > > > > > > > > >> deprecation
> > > > > > > > > > >>>>>>> process
> > > > > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since
> > > Java
> > > > > does
> > > > > > > not
> > > > > > > > > > >>>> allow
> > > > > > > > > > >>>>>> us
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>> extend annotation, I think it would be
> feasible
> > to
> > > > > have
> > > > > > > the
> > > > > > > > > > >> new
> > > > > > > > > > >>>>>>> @Retired
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>> help us monitor and manage the deprecation
> > > process,
> > > > > > house
> > > > > > > > > > >>>>> cleaning,
> > > > > > > > > > >>>>>>> etc.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> @Retired:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> 1. There must be a replacement with
> > functionality
> > > > > > > > > > >> compatibility
> > > > > > > > > > >>>>>> before
> > > > > > > > > > >>>>>>>>> APIs
> > > > > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and
> VOTE
> > > > > > processes
> > > > > > > > > > >> on
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>> ML
> > > > > > > > > > >>>>>>> are
> > > > > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed
> > after 1
> > > > > minor
> > > > > > > > > > >> release
> > > > > > > > > > >>>>>>> sharply
> > > > > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further
> check
> > > > > whether
> > > > > > > it
> > > > > > > > > > >> is
> > > > > > > > > > >>>>>>> possible).
> > > > > > > > > > >>>>>>>>>> Devs who marked them as @Retired are
> responsible
> > > to
> > > > > > remove
> > > > > > > > > > >> them.
> > > > > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and
> @PublicEvolving
> > ->
> > > > > > > @Retired
> > > > > > > > > > >> are
> > > > > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and
> > > @Internal
> > > > > ->
> > > > > > > > > > >> @Retired
> > > > > > > > > > >>>>>> could
> > > > > > > > > > >>>>>>>>> also
> > > > > > > > > > >>>>>>>>>> be used if it can increase user-friendliness
> or
> > > > > > > > > > >>>> dev-friendliness,
> > > > > > > > > > >>>>>> but
> > > > > > > > > > >>>>>>> not
> > > > > > > > > > >>>>>>>>>> mandatory.
> > > > > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired
> to
> > > > > support
> > > > > > > the
> > > > > > > > > > >>>>>>> deprecation
> > > > > > > > > > >>>>>>>>>> process management. Further extension is
> > possible,
> > > > > since
> > > > > > > the
> > > > > > > > > > >>>>>>> annotation
> > > > > > > > > > >>>>>>>>> is
> > > > > > > > > > >>>>>>>>>> built by us.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Best regards,
> > > > > > > > > > >>>>>>>>>> Jing
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the
> > > replies
> > > > > > inline
> > > > > > > > > > >>>> below.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting
> a
> > > > Public
> > > > > > API
> > > > > > > > > > >> is
> > > > > > > > > > >>>>> also a
> > > > > > > > > > >>>>>>>>> kind of
> > > > > > > > > > >>>>>>>>>>>> such change, just like removing one, which
> can
> > > > only
> > > > > > > > > > >> happen
> > > > > > > > > > >>>> with
> > > > > > > > > > >>>>>>> major
> > > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> > > demoting
> > > > > > > Public
> > > > > > > > > > >>>> APIs
> > > > > > > > > > >>>>>>>>> anytime,
> > > > > > > > > > >>>>>>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> > > before
> > > > > > > > > > >> reaching
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > > > demoting
> > > > > > > > > > >> would
> > > > > > > > > > >>>> be a
> > > > > > > > > > >>>>>>> weaker
> > > > > > > > > > >>>>>>>>>>>> change compared to removing the API
> > immediately
> > > > upon
> > > > > > > > > > >> major
> > > > > > > > > > >>>>>> version
> > > > > > > > > > >>>>>>>>> bumps,
> > > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > > > 2-minor-release
> > > > > > > > > > >>>>>> migration
> > > > > > > > > > >>>>>>>>>>> period.
> > > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > > > `@PublicEvolving`
> > > > > > > > > > >> sounds
> > > > > > > > > > >>>>>>> against
> > > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new
> annotation
> > > if
> > > > > > > > > > >> necessary,
> > > > > > > > > > >>>>>> e.g.,
> > > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if
> I
> > > > don't
> > > > > > > > > > >> change my
> > > > > > > > > > >>>>> code
> > > > > > > > > > >>>>>>> at
> > > > > > > > > > >>>>>>>>> all,
> > > > > > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without
> an
> > > > > issue, I
> > > > > > > am
> > > > > > > > > > >> fine
> > > > > > > > > > >>>>>> with
> > > > > > > > > > >>>>>>> all
> > > > > > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > > > > > >>>>>>>>>>> Actually I think there are some dependency
> > > version
> > > > > > > > > > >> resolution
> > > > > > > > > > >>>>>>> policies
> > > > > > > > > > >>>>>>>>> out
> > > > > > > > > > >>>>>>>>>>> there which picks the highest minor version
> > when
> > > > the
> > > > > > > > > > >>>> dependencies
> > > > > > > > > > >>>>>>> pull
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>> multiple minor versions of the same jar,
> which
> > > may
> > > > be
> > > > > > > > > > >> broken if
> > > > > > > > > > >>>>> we
> > > > > > > > > > >>>>>>>>> remove
> > > > > > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > > > "legal"
> > > > > > > that
> > > > > > > > > > >> we
> > > > > > > > > > >>>>> bump
> > > > > > > > > > >>>>>> up
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > > > needed.
> > > > > > > > > > >> However,
> > > > > > > > > > >>>>> this
> > > > > > > > > > >>>>>>> also
> > > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that
> > public
> > > > APIs
> > > > > > > will
> > > > > > > > > > >>>> stay
> > > > > > > > > > >>>>>>> stable
> > > > > > > > > > >>>>>>>>>>>> within the major release series, as the
> series
> > > can
> > > > > end
> > > > > > > > > > >>>> anytime.
> > > > > > > > > > >>>>>>> IMHO,
> > > > > > > > > > >>>>>>>>>>> short
> > > > > > > > > > >>>>>>>>>>>> major release series are not something "make
> > the
> > > > end
> > > > > > > > > > >> users
> > > > > > > > > > >>>>>> happy",
> > > > > > > > > > >>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> > > make
> > > > > > > > > > >> frequent
> > > > > > > > > > >>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > > > approach,
> > > > > > we
> > > > > > > > > > >> can
> > > > > > > > > > >>>>>> still
> > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > >>>>>>>>>>>> longer major release series, while only
> > allowing
> > > > > > Public
> > > > > > > > > > >> APIs
> > > > > > > > > > >>>>>>>>> deprecated
> > > > > > > > > > >>>>>>>>>>> at
> > > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > > > removed
> > > > > in
> > > > > > > > > > >> the
> > > > > > > > > > >>>> next
> > > > > > > > > > >>>>>>> major
> > > > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I totally agree that frequent major version
> > bumps
> > > > are
> > > > > > not
> > > > > > > > > > >>>> ideal,
> > > > > > > > > > >>>>>> but
> > > > > > > > > > >>>>>>>>> here
> > > > > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump
> > > which
> > > > > > > > > > >> removes a
> > > > > > > > > > >>>>>> Public
> > > > > > > > > > >>>>>>>>> API.
> > > > > > > > > > >>>>>>>>>>> So the context is that we have already
> decided
> > to
> > > > > > remove
> > > > > > > > > > >> this
> > > > > > > > > > >>>>>> Public
> > > > > > > > > > >>>>>>> API
> > > > > > > > > > >>>>>>>>>>> while keeping everything else backwards
> > > > compatible. I
> > > > > > > > > > >> think a
> > > > > > > > > > >>>>> major
> > > > > > > > > > >>>>>>>>> version
> > > > > > > > > > >>>>>>>>>>> bump is a commonly understood signal here,
> > > compared
> > > > > > with
> > > > > > > a
> > > > > > > > > > >>>> minor
> > > > > > > > > > >>>>>>> version
> > > > > > > > > > >>>>>>>>>>> change. From end users' perspective, for
> those
> > > who
> > > > > are
> > > > > > > not
> > > > > > > > > > >>>>>> impacted,
> > > > > > > > > > >>>>>>> in
> > > > > > > > > > >>>>>>>>>>> this case upgrading a major version is not
> > > > > necessarily
> > > > > > > more
> > > > > > > > > > >>>>>> involved
> > > > > > > > > > >>>>>>>>> than
> > > > > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as
> > > > smooth
> > > > > > as a
> > > > > > > > > > >>>>>> dependency
> > > > > > > > > > >>>>>>>>>>> version change. For those who are impacted,
> > they
> > > > will
> > > > > > > lose
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> Public
> > > > > > > > > > >>>>>>>>> API
> > > > > > > > > > >>>>>>>>>>> anyways and a major version bump ensures
> there
> > is
> > > > no
> > > > > > > > > > >> surprise.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong
> Song <
> > > > > > > > > > >>>>>> tonysong820@gmail.com <mailto:
> tonysong820@gmail.com
> > >
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Public API is a well defined common concept,
> > and
> > > > one
> > > > > > of
> > > > > > > > > > >> its
> > > > > > > > > > >>>>>>>>>>>>> convention is that it only changes with a
> > major
> > > > > > version
> > > > > > > > > > >>>>> change.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I agree. And from my understanding,
> demoting a
> > > > > Public
> > > > > > > > > > >> API is
> > > > > > > > > > >>>>>> also a
> > > > > > > > > > >>>>>>>>> kind
> > > > > > > > > > >>>>>>>>>>>> of such change, just like removing one,
> which
> > > can
> > > > > only
> > > > > > > > > > >> happen
> > > > > > > > > > >>>>>> with
> > > > > > > > > > >>>>>>>>> major
> > > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> > > demoting
> > > > > > > Public
> > > > > > > > > > >>>> APIs
> > > > > > > > > > >>>>>>>>> anytime,
> > > > > > > > > > >>>>>>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> > > before
> > > > > > > > > > >> reaching
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > > > demoting
> > > > > > > > > > >> would
> > > > > > > > > > >>>> be a
> > > > > > > > > > >>>>>>> weaker
> > > > > > > > > > >>>>>>>>>>>> change compared to removing the API
> > immediately
> > > > upon
> > > > > > > > > > >> major
> > > > > > > > > > >>>>>> version
> > > > > > > > > > >>>>>>>>> bumps,
> > > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > > > 2-minor-release
> > > > > > > > > > >>>>>> migration
> > > > > > > > > > >>>>>>>>>>> period.
> > > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > > > `@PublicEvolving`
> > > > > > > > > > >> sounds
> > > > > > > > > > >>>>>>> against
> > > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new
> annotation
> > > if
> > > > > > > > > > >> necessary,
> > > > > > > > > > >>>>>> e.g.,
> > > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> But it should be
> > > > > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version
> if
> > > we
> > > > > > really
> > > > > > > > > > >> want
> > > > > > > > > > >>>>> to
> > > > > > > > > > >>>>>>> get
> > > > > > > > > > >>>>>>>>> rid
> > > > > > > > > > >>>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's
> completely
> > > > > "legal"
> > > > > > > > > > >> that we
> > > > > > > > > > >>>>>> bump
> > > > > > > > > > >>>>>>> up
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > > > needed.
> > > > > > > > > > >> However,
> > > > > > > > > > >>>>> this
> > > > > > > > > > >>>>>>> also
> > > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that
> > public
> > > > APIs
> > > > > > > will
> > > > > > > > > > >>>> stay
> > > > > > > > > > >>>>>>> stable
> > > > > > > > > > >>>>>>>>>>>> within the major release series, as the
> series
> > > can
> > > > > end
> > > > > > > > > > >>>> anytime.
> > > > > > > > > > >>>>>>> IMHO,
> > > > > > > > > > >>>>>>>>>>> short
> > > > > > > > > > >>>>>>>>>>>> major release series are not something "make
> > the
> > > > end
> > > > > > > > > > >> users
> > > > > > > > > > >>>>>> happy",
> > > > > > > > > > >>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> > > make
> > > > > > > > > > >> frequent
> > > > > > > > > > >>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > > > approach,
> > > > > > we
> > > > > > > > > > >> can
> > > > > > > > > > >>>>>> still
> > > > > > > > > > >>>>>>>>> have
> > > > > > > > > > >>>>>>>>>>>> longer major release series, while only
> > allowing
> > > > > > Public
> > > > > > > > > > >> APIs
> > > > > > > > > > >>>>>>>>> deprecated
> > > > > > > > > > >>>>>>>>>>> at
> > > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > > > removed
> > > > > in
> > > > > > > > > > >> the
> > > > > > > > > > >>>> next
> > > > > > > > > > >>>>>>> major
> > > > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Given our track record I would prefer a
> > regular
> > > > > cycle
> > > > > > > > > > >> (1-2
> > > > > > > > > > >>>>> years)
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic,
> and
> > > not
> > > > > put
> > > > > > > it
> > > > > > > > > > >>>> again
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > > > expectation
> > > > > > > on
> > > > > > > > > > >>>> when
> > > > > > > > > > >>>>>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a
> > good
> > > > > time
> > > > > > > > > > >> for new
> > > > > > > > > > >>>>>> major
> > > > > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking
> > > changes
> > > > > > > > > > >> needed.
> > > > > > > > > > >>>> That
> > > > > > > > > > >>>>>>> makes
> > > > > > > > > > >>>>>>>>> 1-2
> > > > > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic,
> > while
> > > > > > leaving
> > > > > > > > > > >> us
> > > > > > > > > > >>>> more
> > > > > > > > > > >>>>>>> time
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Xintong
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay
> > > Schepler
> > > > <
> > > > > > > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid
> > releasing
> > > > > minor
> > > > > > > > > > >>>>> versions
> > > > > > > > > > >>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>> previous
> > > > > > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not
> > sure
> > > > how
> > > > > > > > > > >> viable
> > > > > > > > > > >>>> that
> > > > > > > > > > >>>>>> is
> > > > > > > > > > >>>>>>> in
> > > > > > > > > > >>>>>>>>>>>>> practice.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially
> > > dropping
> > > > > > > > > > >> support
> > > > > > > > > > >>>> for
> > > > > > > > > > >>>>>> Java
> > > > > > > > > > >>>>>>>>> 8/11,
> > > > > > > > > > >>>>>>>>>>>>> which may very well be a problem for our
> > > current
> > > > > > users.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> > > > > confusion
> > > > > > > > > > >> about
> > > > > > > > > > >>>>>> Public
> > > > > > > > > > >>>>>>>>> API
> > > > > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there
> was a
> > > > > mistake
> > > > > > > > > > >> in
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>> NOTES
> > > > > > > > > > >>>>>>>>> part
> > > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I
> > just
> > > > > fixed
> > > > > > > > > > >> it.
> > > > > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to
> hear
> > > that
> > > > > we
> > > > > > > > > > >> are on
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>> same
> > > > > > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version
> > if
> > > we
> > > > > > > > > > >> really
> > > > > > > > > > >>>> want
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>> get
> > > > > > > > > > >>>>>>>>>>> rid
> > > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it
> took
> > > to
> > > > > get
> > > > > > us
> > > > > > > > > > >> to
> > > > > > > > > > >>>>> 2.0.
> > > > > > > > > > >>>>>> ;)
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on
> > the
> > > > > > cadence
> > > > > > > > > > >> of
> > > > > > > > > > >>>>> major
> > > > > > > > > > >>>>>>>>> releases
> > > > > > > > > > >>>>>>>>>>>>> going forward, and there seem to be
> different
> > > > > > opinions
> > > > > > > > > > >> on
> > > > > > > > > > >>>>> that.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor
> > > releases
> > > > > > > between
> > > > > > > > > > >>>> major
> > > > > > > > > > >>>>>>> ones,
> > > > > > > > > > >>>>>>>>>>>>> that for us means that users have to
> > > potentially
> > > > > deal
> > > > > > > > > > >> with
> > > > > > > > > > >>>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a
> > lot.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a
> > regular
> > > > > cycle
> > > > > > > > > > >> (1-2
> > > > > > > > > > >>>>>> years)
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic,
> and
> > > not
> > > > > put
> > > > > > > it
> > > > > > > > > > >>>> again
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > > > expectation
> > > > > > > on
> > > > > > > > > > >>>> when
> > > > > > > > > > >>>>>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a
> separate
> > > > > thread.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral
> change
> > in
> > > > > mind?
> > > > > > > > > > >> Not
> > > > > > > > > > >>>>> sure
> > > > > > > > > > >>>>>> I
> > > > > > > > > > >>>>>>>>> fully
> > > > > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral
> change
> > > > here.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> > > > > performance
> > > > > > in
> > > > > > > > > > >>>>> certain
> > > > > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> > > > > unknowingly)
> > > > > > > > > > >> relied
> > > > > > > > > > >>>>> upon
> > > > > > > > > > >>>>>>>>>>>>> (
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > > > > > ),
> > > > > > > > > > a semantic change to some
> > > > > > > > > > >> API.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> > > > > submission.
> > > > > > A
> > > > > > > > > > >> few
> > > > > > > > > > >>>>>>> releases
> > > > > > > > > > >>>>>>>>> back
> > > > > > > > > > >>>>>>>>>>>>> we made changes such that the
> initialization
> > of
> > > > the
> > > > > > job
> > > > > > > > > > >>>> master
> > > > > > > > > > >>>>>>>>> happens
> > > > > > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > > > > > >>>>>>>>>>>>> This meant the job submission call returns
> > > > sooner,
> > > > > > and
> > > > > > > > > > >> the
> > > > > > > > > > >>>> job
> > > > > > > > > > >>>>>>> state
> > > > > > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible
> > change,
> > > > but
> > > > > > the
> > > > > > > > > > >>>>> observed
> > > > > > > > > > >>>>>>>>> behavior
> > > > > > > > > > >>>>>>>>>>>>> may be different.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over
> > > time
> > > > we
> > > > > > > > > > >> changed
> > > > > > > > > > >>>>> what
> > > > > > > > > > >>>>>>> some
> > > > > > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Martijn Visser <ma...@apache.org>.
Hi all,

Thanks for the lively and good discussion. Given the length of the
discussion, I skimmed through and then did a deep dive on the latest state
of the FLIP. I think the FLIP is overall in a good state and ready to bring
to a vote.

One thing that I did notice while skimming through the discussions is that
I think there are some follow-ups that could be worthy of a FLIP and a
discussion. For example, if/what/how does the Flink community offer any
behavioral stability guarantees, or other types of expectations and
guarantees. I also do think that we must have tooling in place to implement
this FLIP (and also FLIP-196 and FLIP-197), to avoid that we're not
creating a policy on paper, but also have the means to enforce it. Last but
not least, I echo Jark's point that we can't estimate maintenance cost with
a concrete design and code/POC. For me, that means that a contributor can
choose to propose a deviation, but that the contributor would need to
explicitly mention it in the FLIP and get it discussed/voted on as part of
the FLIP process. But the starting point is as defined in this and other
relevant FLIPs.

Best regards,

Martijn

On Tue, Jun 27, 2023 at 3:38 AM Becket Qin <be...@gmail.com> wrote:

> Hi Xintong, Jark and Jing,
>
> Thanks for the reply. Yes, we can only mark the DataStream API as
> @Deprecated after the ProcessFunction API is fully functional and mature.
>
> It is a fair point that the condition of marking a @Public API as
> deprecated should also be a part of this FLIP. I just added that to the
> FLIP wiki. This is probably more of a clarification on the existing
> convention, rather than a change.
>
> It looks like we are on the same page now for this FLIP. If so, I'll start
> a VOTE thread in two days.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Jun 26, 2023 at 8:09 PM Xintong Song <to...@gmail.com>
> wrote:
>
> > >
> > > Considering DataStream API is the most fundamental and complex API of
> > > Flink, I think it is worth a longer time than the general process for
> the
> > > deprecation period to wait for the new API be mature.
> > >
> >
> > This inspires me. In this specific case, compared to how long should
> > DataStream API be removed after deprecation, it's probably more important
> > to answer the question how long would ProcessFunction API become mature
> and
> > stable after being introduced. According to FLIP-197[1], it requires 4
> > minor releases by default to promote an @Experimental API to @Public. And
> > for ProcessFunction API, which aims to replace DataStream API as one of
> the
> > most fundamental API of Flink, I'd expect this to take at least the
> default
> > time, or even longer. And we probably should wait until we believe
> > ProcessFunction API is stable to mark DataStream API as deprecated,
> rather
> > than as soon as it's introduced. Assuming we introduce the
> ProcessFunction
> > API in 2.0, that means we would need to wait for 6 minor releases (4 for
> > the new API to become stable, and 2 for the migration period) to remove
> > DataStream API, which is ~2.5 year (assuming 5 months / minor release),
> > which sounds acceptable for another major version bump.
> >
> > To wrap things up, it seems to me, sadly, that anyway we cannot avoid the
> > overhead for maintaining both DataStream & ProcessFunction APIs for at
> > least 6 minor releases.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> >
> >
> >
> > On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi all,
> > >
> > > Just want to make sure we are on the same page. There is another
> > example[1]
> > > I was aware of recently that shows why more factors need to be taken
> care
> > > of than just the migration period. Thanks Galen for your hint.
> > >
> > > To put it simply, the concern about API deprecation is not that
> > deprecated
> > > APIs have been removed too early (min migration period is required).
> The
> > > major concern is that APIs are marked as deprecated for a (too) long
> > time,
> > > much longer than the migration period discussed in this thread, afaik.
> > > Since there is no clear picture/definition, no one knows when to do the
> > > migration for users(after the migration period has expired) and when to
> > > remove deprecated APIs for Flink developers.
> > >
> > > Based on all the information I knew, there are two kinds of obstacles
> > that
> > > will and should block the deprecation process:
> > >
> > > 1. Lack of functionalities in new APIs. It happens e.g. with the
> > > SourceFunction to FLIP-27 Source migration. Users who rely on those
> > > functions can not migrate to new APIs.
> > > 2. new APIs have critical bugs. An example could be found at [1]. Users
> > > have to stick to the deprecated APIs.
> > >
> > > Since FLIP-321 is focusing on the API deprecation process, those
> blocking
> > > issues deserve attention and should be put into the FLIP. The current
> > FLIP
> > > seems to only focus on migration periods. If we consider those blocking
> > > issues as orthogonal issues that are beyond the scope of this
> discussion,
> > > does it make sense to change the FLIP title to something like
> "Introduce
> > > minimum migration periods of API deprecation process"?
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
> > >
> > > On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > I agree with Jingsong and Becket.
> > > >
> > > > Look at the legacy SourceFunction (a small part of DataStream API),
> > > > the SourceFunction is still not and can't be marked deprecated[1]
> until
> > > > now after the new Source was released 2 years ago, because the new
> > Source
> > > > still can't fully consume the abilities of legacy API. Considering
> > > > DataStream
> > > > API is the most fundamental and complex API of Flink, I think it is
> > worth
> > > > a longer time than the general process for the deprecation period to
> > > > wait for the new API be mature. The above 2 options sound a bit of
> rush
> > > > for such a widely used API.
> > > >
> > > > I fully understand the concern of maintenance overhead, but it's a
> bit
> > > hard
> > > > for others to estimate maintenance costs without a concrete design
> and
> > > code
> > > > of the new ProcessFunction API. I agree with Becket that maybe we can
> > > > re-evaluate the API deprecation process once we have the new
> > > > ProcessFunction
> > > > API. If the maintenance is indeed huge, I think it is reasonable to
> > have
> > > a
> > > > special rule for this case at that time.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/FLINK-28045
> > > >
> > > > On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com>
> wrote:
> > > >
> > > > > Hi Jingsong,
> > > > >
> > > > > Thanks for the reply. I completely agree with you.
> > > > >
> > > > > The above 2 options are based on the assumption that the community
> > > cannot
> > > > > afford to maintain the deprecated DataStream API for long. I'd say
> we
> > > > > should try everything we can to maintain it for as much time as
> > > possible.
> > > > > DataStream API is actually the most used API in Flink by so many
> > users
> > > at
> > > > > this point. Removing it any time soon will dramatically hurt our
> > users.
> > > > So
> > > > > ideally we should keep it for at least 2 years after deprecation,
> if
> > > not
> > > > > more.
> > > > >
> > > > > The prohibitively high maintenance overhead is just an assumption.
> > > > > Personally speaking, I don't feel this assumption is necessarily
> > true.
> > > We
> > > > > should re-evaluate once we have the new ProcessFunction API in
> place.
> > > > > Without the code it is hard to tell for sure. I am actually kind of
> > > > > optimistic about the maintenance cost.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <
> jingsonglee0@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Becket and all for your discussion.
> > > > > >
> > > > > > > 1. We say this FLIP is enforced starting release 2.0. For
> current
> > > 1.x
> > > > > > APIs,
> > > > > > we provide a migration period with best effort, while allowing
> > > > exceptions
> > > > > > for immediate removal in 2.0. That means we will still try with
> > best
> > > > > effort
> > > > > > to get the ProcessFuncion API ready and deprecate the DataStream
> > API
> > > in
> > > > > > 1.x, but will also be allowed to remove DataStream API in 2.0 if
> > it's
> > > > not
> > > > > > deprecated 2 minor releases before the major version bump.
> > > > > >
> > > > > > > 2. We strictly follow the process in this FLIP, and will
> quickly
> > > bump
> > > > > the
> > > > > > major version from 2.x to 3.0 once the migration period for
> > > DataStream
> > > > > API
> > > > > > is reached.
> > > > > >
> > > > > > Sorry, I didn't read the previous detailed discussion because the
> > > > > > discussion list was so long.
> > > > > >
> > > > > > I don't really like either of these options.
> > > > > >
> > > > > > Considering that DataStream is such an important API, can we
> offer
> > a
> > > > > third
> > > > > > option:
> > > > > >
> > > > > > 3. Maintain the DataStream API throughout 2.X and remove it until
> > > 3.x.
> > > > > But
> > > > > > there's no need to assume that 2.X is a short version, it's
> still a
> > > > > normal
> > > > > > major version.
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> > > > > >
> > > > > > > Thanks much for the input, John, Stefan and Jing.
> > > > > > >
> > > > > > > I think Xingtong has well summarized the pros and cons of the
> two
> > > > > > options.
> > > > > > > Let's collect a few more opinions here and we can move forward
> > with
> > > > the
> > > > > > one
> > > > > > > more people prefer.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Thanks Xingtong for the summary. If I could only choose one
> of
> > > the
> > > > > > given
> > > > > > > > two options, I would go with option 1. I understood that
> > option 2
> > > > > > worked
> > > > > > > > great with Kafka. But the bridge release will still confuse
> > users
> > > > and
> > > > > > my
> > > > > > > > gut feeling is that many users will skip 2.0 and be waiting
> for
> > > 3.0
> > > > > or
> > > > > > > even
> > > > > > > > 3.x. And since fewer users will use Flink 2.x, the
> development
> > > > focus
> > > > > > will
> > > > > > > > be on Flink 3.0 with the fact that the current Flink release
> is
> > > > 1.17
> > > > > > and
> > > > > > > we
> > > > > > > > are preparing 2.0 release. That is weird for me.
> > > > > > > >
> > > > > > > > THB, I would not name the change from @Public to @Retired as
> a
> > > > > > demotion.
> > > > > > > > The purpose of @Retire is to extend the API lifecycle with
> one
> > > more
> > > > > > > stage,
> > > > > > > > like in the real world, people born, studied, graduated,
> > worked,
> > > > and
> > > > > > > > retired. Afaiu from the previous discussion, there are two
> > rules
> > > > we'd
> > > > > > > like
> > > > > > > > to follow simultaneously:
> > > > > > > >
> > > > > > > > 1. Public APIs can only be changed between major releases.
> > > > > > > > 2. A smooth migration phase should be offered to users, i.e.
> at
> > > > > least 2
> > > > > > > > minor releases after APIs are marked as @deprecated. There
> > should
> > > > be
> > > > > > new
> > > > > > > > APIs as the replacement.
> > > > > > > >
> > > > > > > > Agree, those rules are good to improve the user friendliness.
> > > > Issues
> > > > > we
> > > > > > > > discussed are rising because we want to fulfill both of them.
> > If
> > > we
> > > > > > take
> > > > > > > > care of deprecation very seriously, APIs can be marked as
> > > > > @Deprecated,
> > > > > > > only
> > > > > > > > when the new APIs as the replacement provide all
> > functionalities
> > > > the
> > > > > > > > deprecated APIs have. In an ideal case without critical bugs
> > that
> > > > > might
> > > > > > > > stop users adopting the new APIs. Otherwise the expected
> > > > > "replacement"
> > > > > > > will
> > > > > > > > not happen. Users will still stick to the deprecated APIs,
> > > because
> > > > > the
> > > > > > > new
> > > > > > > > APIs can not be used. For big features, it will need at
> least 4
> > > > minor
> > > > > > > > releases(ideal case), i.e. 2+ years to remove deprecated
> APIs:
> > > > > > > >
> > > > > > > > - 1st minor release to build the new APIs as the replacement
> > and
> > > > > > waiting
> > > > > > > > for feedback. It might be difficult to mark the old API as
> > > > deprecated
> > > > > > in
> > > > > > > > this release, because we are not sure if the new APIs could
> > cover
> > > > > 100%
> > > > > > > > functionalities.
> > > > > > > > -  In the lucky case,  mark all old APIs as deprecated in the
> > 2nd
> > > > > minor
> > > > > > > > release. (I would even suggest having the new APIs released
> at
> > > > least
> > > > > > for
> > > > > > > > two minor releases before marking it as deprecated to make
> sure
> > > > they
> > > > > > can
> > > > > > > > really replace the old APIs, in case we care more about
> smooth
> > > > > > migration)
> > > > > > > > - 3rd minor release for the migration period
> > > > > > > > -  In another lucky case, the 4th release is a major release,
> > the
> > > > > > > > deprecated APIs could be removed.
> > > > > > > >
> > > > > > > > The above described scenario works only in an ideal case. In
> > > > reality,
> > > > > > it
> > > > > > > > might take longer to get the new APIs ready and mark the old
> > API
> > > > > > > > deprecated. Furthermore, if the 4th release is not a major
> > > release,
> > > > > we
> > > > > > > will
> > > > > > > > have to maintain both APIs for many further minor releases.
> The
> > > > > > question
> > > > > > > is
> > > > > > > > how to know the next major release in advance, especially 4
> > minor
> > > > > > > releases'
> > > > > > > > period, i.e. more than 2 years in advance? Given that Flink
> > > > contains
> > > > > > many
> > > > > > > > modules, it is difficult to ask devs to create a 2-3 years
> > > > > deprecation
> > > > > > > plan
> > > > > > > > for each case. In case we want to build major releases at a
> > fast
> > > > > pace,
> > > > > > > > let's say every two years, it means devs must plan any API
> > > > > deprecation
> > > > > > > > right after each major release. Afaiac, it is quite
> difficult.
> > > > > > > >
> > > > > > > > The major issue is, afaiu, if we follow rule 2, we have to
> keep
> > > all
> > > > > > > @Public
> > > > > > > > APIs, e.g. DataStream, that are not marked as deprecated yet,
> > to
> > > > 2.0.
> > > > > > > Then
> > > > > > > > we have to follow rule 1 to keep it unchanged until we have
> > 3.0.
> > > > That
> > > > > > is
> > > > > > > > why @Retired is useful to give devs more flexibility and
> still
> > > > > fulfill
> > > > > > > both
> > > > > > > > rules. Let's check it with examples:
> > > > > > > >
> > > > > > > > - we have @Public DataStream API in 1.18. It will not be
> marked
> > > > > > > > as @Deprecated, because the new APIs as the replacement are
> not
> > > > > ready.
> > > > > > > > - we keep the DataStream API itself unchanged in 2.0, but
> > change
> > > > the
> > > > > > > > annotation from @Public to @Retire. New APIs will be
> introduced
> > > > too.
> > > > > In
> > > > > > > > this case, Rule 1 is ok, since public API is allowed to
> change
> > > > > between
> > > > > > > > major releases. Only changing annotation is the minimal
> change
> > we
> > > > > could
> > > > > > > do
> > > > > > > > and it does not break rule 1. Rule 2 is ok too, since the
> > > > DataStream
> > > > > > APIs
> > > > > > > > work exactly the same.  Attention: the change of @Public ->
> > > > @Retired
> > > > > > can
> > > > > > > > only be done between major releases, because @Public APIs can
> > > only
> > > > be
> > > > > > > > changed between major releases.
> > > > > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > > > > - in 2.3, DataStream will be removed.
> > > > > > > >
> > > > > > > > Becket mentioned previously (please correct me if I didn't
> > > > understand
> > > > > > it
> > > > > > > > correctly) that users might not check the changes of
> annotation
> > > and
> > > > > the
> > > > > > > > upgrade process might not be smooth. I was wondering, if
> users
> > > > don't
> > > > > > pay
> > > > > > > > attention to that, they will also not pay attention to
> > > @deprecated.
> > > > > The
> > > > > > > > migration will not be smooth too even when we stick to the
> Java
> > > > > > > > standard @deprecated. Let's go throw it with examples:
> > > > > > > >
> > > > > > > > Scenario 1: with @Retired
> > > > > > > >
> > > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing
> > should
> > > > be
> > > > > > done
> > > > > > > > for users who still want to use DataStream. The migration is
> > > > smooth.
> > > > > > But
> > > > > > > > they should be aware of the annotation changes, i.e. @Public
> ->
> > > > > > @Retired
> > > > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is
> removed.
> > > > Since
> > > > > > it
> > > > > > > is
> > > > > > > > a major release upgrade, breaking changes are expected.
> > > > > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is
> removed.
> > > > Since
> > > > > > > APIs
> > > > > > > > are marked as @Retired, breaking changes are expected too.
> > > > > > > >
> > > > > > > > In addition to fulfilling both rules, there are two more
> > benefits
> > > > of
> > > > > > this
> > > > > > > > option:
> > > > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > > > > > 2. no bridge release is required. DataStream API could be
> > > > deprecated
> > > > > in
> > > > > > > 2.x
> > > > > > > > without breaking any of the two rules. @Retired provides
> enough
> > > > > > > flexibility
> > > > > > > > for future deprecations and upgrades.
> > > > > > > >
> > > > > > > > The only issue I can find is that users might be surprised in
> > > case
> > > > 3,
> > > > > > if
> > > > > > > > they ignore the previous @Public -> @Retired change. But for
> > such
> > > > > > users,
> > > > > > > > they will ignore @deprecated annotation too and will have
> > issues
> > > > with
> > > > > > > > pure @deprecated
> > > > > > > > annotation too:
> > > > > > > >
> > > > > > > > Scenario 2: with @deprecated
> > > > > > > >
> > > > > > > > The example will be slightly different from scenario 1,
> because
> > > > > @Public
> > > > > > > > APIs can only be removed between major releases. As an
> example,
> > > > > > > DataStream
> > > > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and
> > then
> > > > > > removed
> > > > > > > in
> > > > > > > > 2.0
> > > > > > > >
> > > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not
> > marked
> > > > as
> > > > > > > > deprecated are removed between major releases. Rule 1 is ok.
> > The
> > > 2
> > > > > > minor
> > > > > > > > releases period defined in rule 2 sounds like a smooth
> > migration
> > > > > plan,
> > > > > > > but
> > > > > > > > for users in this case, it is still not smooth.
> > > > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade
> > process
> > > > > with
> > > > > > > > deprecated APIs. But for those users who ignore annotation
> > > changes,
> > > > > > will
> > > > > > > > have the same issue mentioned in case 1.
> > > > > > > >
> > > > > > > > I understood that, compared to scenario 1, users, who ignore
> > > > @Public
> > > > > > > > -> @Retired
> > > > > > > > changes, will not face any breaking changes, after they
> > migrated
> > > to
> > > > > > 2.0.
> > > > > > > > But the cost of it is bridge releases. There might be many
> > bridge
> > > > > > > releases
> > > > > > > > in the future. As I mentioned previously, it is not easy to
> > find
> > > > the
> > > > > > > right
> > > > > > > > timing to kick off the deprecation process of APIs before the
> > > next
> > > > > > > expected
> > > > > > > > major release and 2-3 years in advance.
> > > > > > > >
> > > > > > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > > > > > @deprecated
> > > > > > > > annotation changes
> > > > > > > >
> > > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20,
> removed
> > in
> > > > > 1.21
> > > > > > > >
> > > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no
> > > > smooth
> > > > > > > > migration.
> > > > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking
> change,
> > > > > > > > since @deprecated is ignored
> > > > > > > >
> > > > > > > > In summary, for users who care about the annotation change,
> > they
> > > > will
> > > > > > get
> > > > > > > > more benefits with @Retired. For users who ignore annotation
> > > > changes,
> > > > > > > they
> > > > > > > > will be always facing non-smooth upgrades even using pure
> > > standard
> > > > > > > > @deprecated
> > > > > > > > annotation/process.
> > > > > > > >
> > > > > > > > For the case that IDEs support @deprecated, that is true.
> There
> > > are
> > > > > no
> > > > > > > such
> > > > > > > > supports for homebuilt annotation. But the intention is to
> let
> > > > users
> > > > > be
> > > > > > > > aware of it. There will be other ways to do it for users who
> > care
> > > > > about
> > > > > > > > annotation changes. For users who do not care, there should
> be
> > no
> > > > > > > > difference between with tool support or without. Using
> @Retired
> > > > with
> > > > > > > > @deprecation
> > > > > > > > together is at least not worse than only using @deprecation
> > > alone.
> > > > > And
> > > > > > > > there are additional benefits with @Retired.
> > > > > > > >
> > > > > > > > Just my 2 cents and looking forward to your feedback,
> > especially
> > > > > > > different
> > > > > > > > opinions that will help me understand the issue better.
> Thanks!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > > > > <sr...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Xintong,
> > > > > > > > >
> > > > > > > > > Thanks for the summary, most of the points that you agreed
> > upon
> > > > > also
> > > > > > > make
> > > > > > > > > sense to me.
> > > > > > > > >
> > > > > > > > > >   2. Dropping deprecated Public APIs in minor releases,
> or
> > > > > demoting
> > > > > > > > APIs
> > > > > > > > > >   from Public to PublicEvolving / Experimental / Retired
> in
> > > any
> > > > > > > version
> > > > > > > > > bump,
> > > > > > > > > >   are not good practices.
> > > > > > > > >
> > > > > > > > > I hope we can can go beyond calling it “not a good
> practice"
> > > and
> > > > > > reach
> > > > > > > > > consensus that we will not demote or remove public APIs in
> > > minor
> > > > > > > releases
> > > > > > > > > and (technically) enforce this rule for all contributions.
> I
> > > > think
> > > > > > > it’s a
> > > > > > > > > reasonable expectation for stability from a project as
> mature
> > > as
> > > > > > Flink.
> > > > > > > > >
> > > > > > > > > > I'm personally in favor of option 1. As the
> > migration-period
> > > > rule
> > > > > > is
> > > > > > > > > newly
> > > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > > where
> > > > we
> > > > > > > > already
> > > > > > > > > > missed the best chance for planning the deprecation.
> > > Moreover,
> > > > I
> > > > > do
> > > > > > > > > believe
> > > > > > > > > > having a quick major version bump does confuse users.
> Yes,
> > we
> > > > can
> > > > > > > > explain
> > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> > anything
> > > > > other
> > > > > > > than
> > > > > > > > > the
> > > > > > > > > > removal of an deprecated API. But having to explain this
> > > itself
> > > > > is
> > > > > > an
> > > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > > understanding,
> > > > > > > his
> > > > > > > > > > major point is that a quick major version bump causes
> > barely
> > > > any
> > > > > > > actual
> > > > > > > > > > lose on users, while in option 1 not providing the
> > migration
> > > > > period
> > > > > > > or
> > > > > > > > > > providing a shorter on is an actual lose on users.
> > (@Becket,
> > > > > please
> > > > > > > > > correct
> > > > > > > > > > me if I'm mistaken.)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I would be open to both options and maybe postpone this
> > > decision
> > > > > > until
> > > > > > > we
> > > > > > > > > have the complete roadmap for Flink 2.0. We would have a
> > better
> > > > > > > overview
> > > > > > > > > about the technical consequences, e.g. how hard it would be
> > to
> > > > > offer
> > > > > > > and
> > > > > > > > > maintain both APIs side by side, how well we get both APIs
> > > > > separated,
> > > > > > > and
> > > > > > > > > whether or not we will be able to use the full potential of
> > our
> > > > > > > breaking
> > > > > > > > > changes before the DataStream API is completely removed.
> One
> > > > > example
> > > > > > > for
> > > > > > > > > the last point I can think of would be moving from lazy to
> > > eager
> > > > > > state
> > > > > > > > > declaration. If we believe that it is reasonable to support
> > all
> > > > > > planned
> > > > > > > > > changes while maintaining both APIs, I'm leaning towards
> > option
> > > > 2.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stefan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <
> > > tonysong820@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Becket and I had an offline voice call earlier today. We
> > have
> > > > > > reached
> > > > > > > > > > consensus on some of the arguments, but still hold
> > different
> > > > > > opinions
> > > > > > > > on
> > > > > > > > > > some others. I'd like to post the outcome here for
> > > > transparency.
> > > > > > > > > >
> > > > > > > > > > We both agree that:
> > > > > > > > > >
> > > > > > > > > >   1. Providing a migration period would be beneficial for
> > our
> > > > > > users,
> > > > > > > > and
> > > > > > > > > >   we should do that
> > > > > > > > > >   2. Dropping deprecated Public APIs in minor releases,
> or
> > > > > demoting
> > > > > > > > APIs
> > > > > > > > > >   from Public to PublicEvolving / Experimental / Retired
> in
> > > any
> > > > > > > version
> > > > > > > > > bump,
> > > > > > > > > >   are not good practices.
> > > > > > > > > >   3. Ideally, with this FLIP, developers should be more
> > > careful
> > > > > and
> > > > > > > > plan
> > > > > > > > > >   API changes ahead. That means:
> > > > > > > > > >      1. Be more careful with designing APIs and promoting
> > > them
> > > > to
> > > > > > > > Public,
> > > > > > > > > >      so that they won't be changed / removed very soon,
> and
> > > > also
> > > > > > the
> > > > > > > > > >      maintenance overhead for keeping them after
> > deprecation
> > > > > should
> > > > > > > be
> > > > > > > > > >      affordable. (I believe this also aligns with John's
> > > > > opinion.)
> > > > > > > > > >      2. Plan the deprecation / removal of old APIs that
> we
> > > > don't
> > > > > > want
> > > > > > > > to
> > > > > > > > > >      carry to the next major version earlier, so that
> > they'll
> > > > > have
> > > > > > > the
> > > > > > > > > >      2-minor-release migration period before the next
> major
> > > > > version
> > > > > > > > bump.
> > > > > > > > > >   4. A practical situation is that, we are not ready for
> > > > > > deprecating
> > > > > > > > the
> > > > > > > > > >   DataStream API, nor afford to carry it for another
> couple
> > > of
> > > > > > years.
> > > > > > > > > >      1. We cannot deprecate it now, because the new
> > > > > ProcessFunction
> > > > > > > API
> > > > > > > > > >      (the planned replacement for DataStream API) is not
> > yet
> > > > > ready.
> > > > > > > We
> > > > > > > > > haven't
> > > > > > > > > >      planned the ProcessFunction API earlier because the
> > > > original
> > > > > > > > > > plan was to do
> > > > > > > > > >      in-place changes directly on the DataStream API in
> > > release
> > > > > > 2.0,
> > > > > > > > > until the
> > > > > > > > > >      smooth migration period is proposed. Before this,
> the
> > > only
> > > > > > > > > guarantees we
> > > > > > > > > >      provide for Public APIs is that they'll stay
> > compatible
> > > > > until
> > > > > > > the
> > > > > > > > > next
> > > > > > > > > >      major version, which led to the understanding the
> > > in-place
> > > > > > > changes
> > > > > > > > > on
> > > > > > > > > >      Public APIs should be fine.
> > > > > > > > > >      2. We also cannot afford carrying the DataStream API
> > for
> > > > > > another
> > > > > > > > > >      couple of years. One of the reasons that we want to
> > > > > refactor /
> > > > > > > > > replace
> > > > > > > > > >      DataStream API is that it exposes and depends on too
> > > many
> > > > > > > > > > Flink's internal
> > > > > > > > > >      runtime implementations, which has already limited /
> > > > blocked
> > > > > > > plans
> > > > > > > > > for
> > > > > > > > > >      Flink's internal improvements for many times. Also
> due
> > > to
> > > > > the
> > > > > > > wide
> > > > > > > > > and
> > > > > > > > > >      transitive dependencies on the internals,
> maintaining
> > > two
> > > > > sets
> > > > > > > of
> > > > > > > > > lower
> > > > > > > > > >      APIs itself is expensive and would be better to keep
> > it
> > > > > short.
> > > > > > > > This
> > > > > > > > > is
> > > > > > > > > >      admittedly a historical design flaw which should be
> > > > avoided
> > > > > in
> > > > > > > > > future.
> > > > > > > > > >
> > > > > > > > > > What we haven't reached consensus is about how to deal
> with
> > > the
> > > > > > > current
> > > > > > > > > > situation around the DataStream API. There are two
> options.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   1. We say this FLIP is enforced starting release 2.0.
> For
> > > > > current
> > > > > > > 1.x
> > > > > > > > > >   APIs, we provide a migration period with best effort,
> > while
> > > > > > > allowing
> > > > > > > > > >   exceptions for immediate removal in 2.0. That means we
> > will
> > > > > still
> > > > > > > try
> > > > > > > > > with
> > > > > > > > > >   best effort to get the ProcessFuncion API ready and
> > > deprecate
> > > > > the
> > > > > > > > > >   DataStream API in 1.x, but will also be allowed to
> remove
> > > > > > > DataStream
> > > > > > > > > API in
> > > > > > > > > >   2.0 if it's not deprecated 2 minor releases before the
> > > major
> > > > > > > version
> > > > > > > > > bump.
> > > > > > > > > >   2. We strictly follow the process in this FLIP, and
> will
> > > > > quickly
> > > > > > > bump
> > > > > > > > > >   the major version from 2.x to 3.0 once the migration
> > period
> > > > for
> > > > > > > > > DataStream
> > > > > > > > > >   API is reached.
> > > > > > > > > >
> > > > > > > > > > I'm personally in favor of option 1. As the
> > migration-period
> > > > rule
> > > > > > is
> > > > > > > > > newly
> > > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > > where
> > > > we
> > > > > > > > already
> > > > > > > > > > missed the best chance for planning the deprecation.
> > > Moreover,
> > > > I
> > > > > do
> > > > > > > > > believe
> > > > > > > > > > having a quick major version bump does confuse users.
> Yes,
> > we
> > > > can
> > > > > > > > explain
> > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> > anything
> > > > > other
> > > > > > > than
> > > > > > > > > the
> > > > > > > > > > removal of an deprecated API. But having to explain this
> > > itself
> > > > > is
> > > > > > an
> > > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > > understanding,
> > > > > > > his
> > > > > > > > > > major point is that a quick major version bump causes
> > barely
> > > > any
> > > > > > > actual
> > > > > > > > > > lose on users, while in option 1 not providing the
> > migration
> > > > > period
> > > > > > > or
> > > > > > > > > > providing a shorter on is an actual lose on users.
> > (@Becket,
> > > > > please
> > > > > > > > > correct
> > > > > > > > > > me if I'm mistaken.)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > And we'd like to hear more feedback from the community.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Xintong
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > > > > vvcephei@apache.org
> > > > > > > > > <ma...@apache.org>> wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Becket and Xintong,
> > > > > > > > > >>
> > > > > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > > > > >>
> > > > > > > > > >> Once an API is marked Public, we're committing to
> support
> > it
> > > > > until
> > > > > > > > it's
> > > > > > > > > >> deprecated, and once it's deprecated, to leave it in
> place
> > > for
> > > > > at
> > > > > > > > least
> > > > > > > > > two
> > > > > > > > > >> minor releases and then only remove it in the following
> > > major
> > > > > > > release.
> > > > > > > > > >>
> > > > > > > > > >> As a user, I would be dismayed to discover that the
> > project
> > > > > > > > maintainers
> > > > > > > > > >> demoted a Public API to PublicEvolving or Experimental
> in
> > > > order
> > > > > to
> > > > > > > > > violate
> > > > > > > > > >> the spirit of the deprecation period for Public APIs.
> > > > Likewise,
> > > > > > I'd
> > > > > > > > > view a
> > > > > > > > > >> rapid sequence of minor releases for the purpose of
> > dropping
> > > > > > > > deprecated
> > > > > > > > > >> APIs in the next major release as another violation of
> the
> > > > > spirit
> > > > > > of
> > > > > > > > our
> > > > > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > > > > >>
> > > > > > > > > >> I'm glad we're gaming out these situations before we
> > > institute
> > > > > > this
> > > > > > > > > FLIP,
> > > > > > > > > >> and I hope we all converge on a clear understanding of
> > what
> > > > > we're
> > > > > > > > > >> guaranteeing. From where I'm sitting, the essence of
> this
> > > FLIP
> > > > > is
> > > > > > > > > >> specifically to prevent us from doing whatever we want
> and
> > > > > instead
> > > > > > > to
> > > > > > > > > >> commit to providing a suitable notice period to users
> who
> > > rely
> > > > > on
> > > > > > > > Public
> > > > > > > > > >> APIs.
> > > > > > > > > >>
> > > > > > > > > >> It honestly shouldn't matter what the maintenance
> overhead
> > > is
> > > > > (or,
> > > > > > > > > rather,
> > > > > > > > > >> evaluating the maintenance overhead should be a
> > > consideration
> > > > > > before
> > > > > > > > we
> > > > > > > > > >> promote an API to Public to begin with). My experience
> in
> > > > other
> > > > > > > > > projects is
> > > > > > > > > >> that regretting a feature so much you want to drop it
> asap
> > > is
> > > > > > > > extremely
> > > > > > > > > >> rare, and that living with the pain for a little while
> > until
> > > > the
> > > > > > > > > >> deprecation period expires will probably make us better
> > > > > engineers.
> > > > > > > > > >>
> > > > > > > > > >> With that all said, one thing that is not a violation is
> > to
> > > > > > propose
> > > > > > > a
> > > > > > > > > new
> > > > > > > > > >> major release in order to drop especially burdensome
> > > > deprecated
> > > > > > APIs
> > > > > > > > > that
> > > > > > > > > >> have already enjoyed their full deprecation period.
> Those
> > > > really
> > > > > > bad
> > > > > > > > > "case
> > > > > > > > > >> A" features will be very rare.
> > > > > > > > > >>
> > > > > > > > > >> Looking over the FLIP again, I see a reference to the
> > > release
> > > > > > > > > >> documentation, but not to the build tooling support for
> > the
> > > > > > > Deprecated
> > > > > > > > > >> annotation, so it might help to share this scenario: The
> > way
> > > > > this
> > > > > > > will
> > > > > > > > > all
> > > > > > > > > >> come together in practice is that, by giving good
> > stability
> > > > > > > > guarantees,
> > > > > > > > > we
> > > > > > > > > >> will encourage our users to bump minor versions when
> they
> > > are
> > > > > > > > released,
> > > > > > > > > >> which means that they will see compiler warnings
> > immediately
> > > > if
> > > > > > any
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> APIs they've used are deprecated. In fact, I'd recommend
> > for
> > > > > them
> > > > > > to
> > > > > > > > use
> > > > > > > > > >> the `-Werror` javac flag to be sure that they don't miss
> > > those
> > > > > > > > warnings.
> > > > > > > > > >> Given our release cadence, this means that they will
> have
> > > > about
> > > > > > six
> > > > > > > > > months
> > > > > > > > > >> to migrate off of any deprecated APIs without being
> forced
> > > to
> > > > > > avoid
> > > > > > > > > >> updates. And they really need to be able to adopt
> further
> > > > minor
> > > > > > > > updates
> > > > > > > > > in
> > > > > > > > > >> the series, since they may contain further improvements
> > that
> > > > > > > actually
> > > > > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > > > > >>
> > > > > > > > > >> The compiler warnings are a far more reliable mechanism
> to
> > > > > > advertise
> > > > > > > > > >> deprecation than release notes, since users' own builds
> > will
> > > > > > notify
> > > > > > > > them
> > > > > > > > > >> right away about only the deprecations that are relevant
> > to
> > > > > them,
> > > > > > > but
> > > > > > > > it
> > > > > > > > > >> does rely on users feeling ok to bump up minor versions
> > in a
> > > > > > timely
> > > > > > > > > >> fashion. If we consistently burn them by not observing
> the
> > > > > > stability
> > > > > > > > > >> guarantees, they'll try to avoid updates instead, and
> the
> > > > whole
> > > > > > > thing
> > > > > > > > > falls
> > > > > > > > > >> apart.
> > > > > > > > > >>
> > > > > > > > > >> Thanks again,
> > > > > > > > > >> -John
> > > > > > > > > >>
> > > > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > > > > >>> Hi Xintong,
> > > > > > > > > >>>
> > > > > > > > > >>> Let's compare the following cases:
> > > > > > > > > >>>
> > > > > > > > > >>> A. If the maintenance overhead of the deprecated API is
> > > high,
> > > > > and
> > > > > > > we
> > > > > > > > > want
> > > > > > > > > >>> to remove it after two minor releases. Then there are
> two
> > > > > > options:
> > > > > > > > > >>>    A1: Demote the API in the major version bump and
> > remove
> > > > the
> > > > > > code
> > > > > > > > > >> with a
> > > > > > > > > >>> minor version bump.
> > > > > > > > > >>>    A2: Do exactly the same as 1-A, except that when
> > > removing
> > > > > the
> > > > > > > > code,
> > > > > > > > > >>> bump up the major version. So this might be a short
> major
> > > > > > version.
> > > > > > > > > >>>
> > > > > > > > > >>> B. If the maintenance overhead of the deprecated API is
> > not
> > > > > high,
> > > > > > > > > >> therefore
> > > > > > > > > >>> keeping it for a long time is affordable. There are
> also
> > > two
> > > > > > > options:
> > > > > > > > > >>>    B1: Same as A-1, demote the API in the major version
> > > bump
> > > > > and
> > > > > > > > remove
> > > > > > > > > >>> the code with a minor version bump.
> > > > > > > > > >>>    B2: Keep the API for all the minor versions in the
> > major
> > > > > > > version,
> > > > > > > > > and
> > > > > > > > > >>> only remove the code in the next major version.
> > > > > > > > > >>>
> > > > > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > > > > >>>
> > > > > > > > > >>> For case A:
> > > > > > > > > >>> The following stays the same for A1 and A2:
> > > > > > > > > >>>  - users will lose the API after two minor leases. So
> the
> > > > > > migration
> > > > > > > > > >> period
> > > > > > > > > >>> is the same for A-1 and A-2.
> > > > > > > > > >>>  - A1 and A2 will have exactly the same code after the
> > > > removal
> > > > > of
> > > > > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > > > > >>>  - To move forward, users need to move to the next
> minor
> > > > > version
> > > > > > in
> > > > > > > > A1,
> > > > > > > > > >> or
> > > > > > > > > >>> to the next major version in A2. Because the code are
> the
> > > > same,
> > > > > > the
> > > > > > > > > >> actual
> > > > > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > > > > >>>
> > > > > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > > > > >>>  - A1 allows keeping the major version release cadence.
> > A-2
> > > > > will
> > > > > > > > have a
> > > > > > > > > >>> short major version release.
> > > > > > > > > >>>  - A1 breaks the well understood API semantic, while
> A-2
> > > does
> > > > > > not.
> > > > > > > > > >>>
> > > > > > > > > >>> From what I see, since there is no well establish
> > standard
> > > > > > > regarding
> > > > > > > > > how
> > > > > > > > > >>> long a major version should be released, A short major
> > > > version
> > > > > > > > release
> > > > > > > > > is
> > > > > > > > > >>> potential and emotional. It is not ideal but does not
> > have
> > > > > > material
> > > > > > > > > >>> downsides compared with A1. I did not hear anyone
> > > complaining
> > > > > > about
> > > > > > > > > Kafka
> > > > > > > > > >>> only has two 1.x release. However, A1 actually breaks
> the
> > > > well
> > > > > > > > > understood
> > > > > > > > > >>> API semantic, which has more material impact.
> > > > > > > > > >>>
> > > > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should
> > > fall
> > > > > into
> > > > > > > > case
> > > > > > > > > B.
> > > > > > > > > >>> So, short major versions should be very occasional. I'd
> > be
> > > > very
> > > > > > > > > concerned
> > > > > > > > > >>> if the reason we choose A1 is simply because we cannot
> > > afford
> > > > > > > > > >> maintaining a
> > > > > > > > > >>> bunch of deprecated APIs until the next major version.
> > This
> > > > > > > indicates
> > > > > > > > > >> that
> > > > > > > > > >>> the actual problem we need to solve is to lower the
> > > > maintenance
> > > > > > > > > overhead
> > > > > > > > > >> of
> > > > > > > > > >>> deprecated APIs, so that we are comfortable to keep
> them
> > > > > longer.
> > > > > > As
> > > > > > > > > John
> > > > > > > > > >>> and I mentioned earlier, there are ways to achieve this
> > and
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > >>> learn how to do it in Flink. Otherwise, our discussion
> > > about
> > > > > > > > versioning
> > > > > > > > > >>> here does not bring much value, because we will end up
> > > with a
> > > > > > bunch
> > > > > > > > of
> > > > > > > > > >>> short-lived APIs which upset our users, no matter how
> we
> > > > > version
> > > > > > > the
> > > > > > > > > >>> releases.
> > > > > > > > > >>>
> > > > > > > > > >>> So, if there are concrete examples that you think will
> > > block
> > > > us
> > > > > > > from
> > > > > > > > > >>> keeping API stability with affordable cost, let's take
> a
> > > look
> > > > > > > > together
> > > > > > > > > >> and
> > > > > > > > > >>> see if that can be improved.
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks,
> > > > > > > > > >>>
> > > > > > > > > >>> Jiangjie (Becket) Qin
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> From what I see,
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > > > > tonysong820@gmail.com
> > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> The part I don't understand is if we are willing to
> > have
> > > a
> > > > > > > > migration
> > > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > > what
> > > > do
> > > > > > we
> > > > > > > > > >> lose to
> > > > > > > > > >>>> do
> > > > > > > > > >>>>> a major version bump instead, so we don't break the
> > > common
> > > > > > > > versioning
> > > > > > > > > >>>>> semantic?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think we are talking about the cases where a major
> > > version
> > > > > > bump
> > > > > > > > > >> happens
> > > > > > > > > >>>> before a deprecated Public API reaches its migration
> > > period.
> > > > > So
> > > > > > > > > >> removing
> > > > > > > > > >>>> the API with another major version bump means we have
> > two
> > > > > > > > consecutive
> > > > > > > > > >> major
> > > > > > > > > >>>> versions in a very short time. And as previously
> > > mentioned,
> > > > > > having
> > > > > > > > > >> frequent
> > > > > > > > > >>>> major version bumps would weaken the value of the
> > > commitment
> > > > > > > "Public
> > > > > > > > > >> API
> > > > > > > > > >>>> stay compatible within a major version". I think users
> > > also
> > > > > have
> > > > > > > > > >>>> expectations about how long a major version should
> live,
> > > > which
> > > > > > > > should
> > > > > > > > > >> be at
> > > > > > > > > >>>> least a couple of years rather than 1-2 minor
> releases.
> > > > > > > > > >>>>
> > > > > > > > > >>>> This is another option, but I think it is very likely
> > more
> > > > > > > expensive
> > > > > > > > > >> than
> > > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > > will
> > > > be
> > > > > > > > > >> questions
> > > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> > in
> > > > > 2.0"?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Yes, it's more expensive than simply bumping the major
> > > > > version,
> > > > > > > but
> > > > > > > > > >> IMHO
> > > > > > > > > >>>> it's probably cheaper than carrying the API until the
> > next
> > > > > major
> > > > > > > > > >> version if
> > > > > > > > > >>>> we don't bump the major version very soon. And see my
> > > reply
> > > > > > above
> > > > > > > on
> > > > > > > > > >> why
> > > > > > > > > >>>> not bumping immediately.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Xintong
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > > > > becket.qin@gmail.com
> > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Hi Xintong,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Please see the replies below.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I see your point, that users would feel surprised if
> > they
> > > > > find
> > > > > > > > > >> things no
> > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> > release.
> > > > > > > However,
> > > > > > > > > >> I'd
> > > > > > > > > >>>>> like
> > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > > similar
> > > > > > > > > >> problem
> > > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > > attention
> > > > > > and
> > > > > > > > > >> make
> > > > > > > > > >>>>> sure
> > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> > longer
> > > > be
> > > > > > > Public
> > > > > > > > > >> in
> > > > > > > > > >>>>> 2.0.
> > > > > > > > > >>>>>> There're various ways to do that, e.g., release
> notes,
> > > > > > warnings
> > > > > > > in
> > > > > > > > > >>>> logs,
> > > > > > > > > >>>>>> etc.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> First of all, I am not a fan of removing
> PublicEvolving
> > > > APIs
> > > > > in
> > > > > > > > minor
> > > > > > > > > >>>>> version changes. Personally speaking, I want them to
> be
> > > > also
> > > > > > > > removed
> > > > > > > > > >> in
> > > > > > > > > >>>>> major version changes.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Empirically, I rarely see projects with complex API
> > > > semantic
> > > > > > work
> > > > > > > > > >> well.
> > > > > > > > > >>>>> Many, if not most, users don't read docs / release
> > notes
> > > /
> > > > > > > warnings
> > > > > > > > > >> in
> > > > > > > > > >>>>> logs. I don't think that is their fault. Typically an
> > > > > > application
> > > > > > > > > >>>> developer
> > > > > > > > > >>>>> will have to deal with dozens of libraries maintained
> > by
> > > > > > various
> > > > > > > > > >>>>> communities / groups. It is just too difficult for
> them
> > > to
> > > > > keep
> > > > > > > > > >> track of
> > > > > > > > > >>>>> all the specific semantics each project puts there.
> If
> > > you
> > > > > > think
> > > > > > > > > >> about
> > > > > > > > > >>>> it,
> > > > > > > > > >>>>> how many of Flink developers read all the release
> notes
> > > of
> > > > > > Guava,
> > > > > > > > > >> Apache
> > > > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you
> upgrade a
> > > > > version
> > > > > > > of
> > > > > > > > > >> them?
> > > > > > > > > >>>>> One would probably try bumping the dependency version
> > and
> > > > see
> > > > > > if
> > > > > > > > > >> there is
> > > > > > > > > >>>>> an exception and solve them case by case. And if it
> is
> > a
> > > > > minor
> > > > > > > > > >> version
> > > > > > > > > >>>>> bump, one probably does not expect an exception at
> all.
> > > > > Another
> > > > > > > > > >> example
> > > > > > > > > >>>> is
> > > > > > > > > >>>>> that in Flink we still have so many usage of
> deprecated
> > > > > methods
> > > > > > > all
> > > > > > > > > >> over
> > > > > > > > > >>>>> the place, and I strongly doubt everyone knows when
> the
> > > > > source
> > > > > > > code
> > > > > > > > > >> of
> > > > > > > > > >>>>> these methods are deprecated and when they should be
> > > > removed.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> So, the most important principle of API is simple and
> > > > > > intuitive.
> > > > > > > > The
> > > > > > > > > >>>>> versioning semantic is a simple and universally
> > accepted
> > > > API
> > > > > > > > > >> stability
> > > > > > > > > >>>>> standard. If we ourselves as Flink developers are
> > relying
> > > > on
> > > > > > this
> > > > > > > > > >> for our
> > > > > > > > > >>>>> own dependencies. I don't think we can expect more
> from
> > > our
> > > > > > > users.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Another possible alternative: whenever there's a
> > > deprecated
> > > > > > > Public
> > > > > > > > > >> API
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>> reaches a major version bump before the migration
> > > period,
> > > > > and
> > > > > > we
> > > > > > > > > >> also
> > > > > > > > > >>>>> don't
> > > > > > > > > >>>>>> want to carry it for all the next major release
> > series,
> > > we
> > > > > may
> > > > > > > > > >> consider
> > > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > > version
> > > > > > > after
> > > > > > > > > >> the
> > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > > then
> > > > we
> > > > > > > bump
> > > > > > > > > >> to
> > > > > > > > > >>>> 2.0,
> > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > > provide
> > > > > > > users
> > > > > > > > > >>>> another
> > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while
> satisfying
> > > the
> > > > > > > > > >>>> 2-minor-release
> > > > > > > > > >>>>>> migration period.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> This is another option, but I think it is very likely
> > > more
> > > > > > > > expensive
> > > > > > > > > >> than
> > > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > > will
> > > > be
> > > > > > > > > >> questions
> > > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> > in
> > > > > 2.0"?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I think my major point is, we should not carry APIs
> > > > > deprecated
> > > > > > > in a
> > > > > > > > > >>>>>> previous major version along all the next major
> > version
> > > > > > series.
> > > > > > > > I'd
> > > > > > > > > >>>> like
> > > > > > > > > >>>>> to
> > > > > > > > > >>>>>> try giving users more commitments, i.e. the
> migration
> > > > > period,
> > > > > > as
> > > > > > > > > >> long
> > > > > > > > > >>>> as
> > > > > > > > > >>>>> it
> > > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> > it
> > > > > > doesn't
> > > > > > > > > >> work,
> > > > > > > > > >>>> I'd
> > > > > > > > > >>>>>> be in favor of not providing the migration period,
> but
> > > > > > fallback
> > > > > > > to
> > > > > > > > > >> only
> > > > > > > > > >>>>>> guarantee the compatibility within the major
> version.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> The part I don't understand is if we are willing to
> > have
> > > a
> > > > > > > > migration
> > > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > > what
> > > > do
> > > > > > we
> > > > > > > > > >> lose to
> > > > > > > > > >>>> do
> > > > > > > > > >>>>> a major version bump instead, so we don't break the
> > > common
> > > > > > > > versioning
> > > > > > > > > >>>>> semantic?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thanks,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > > > > tonysong820@gmail.com
> > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> As an end user who only uses Public APIs, if I
> don't
> > > > change
> > > > > > my
> > > > > > > > > >> code
> > > > > > > > > >>>> at
> > > > > > > > > >>>>>>> all, my expectation is the following:
> > > > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > issue, I
> > > > am
> > > > > > fine
> > > > > > > > > >> with
> > > > > > > > > >>>>> all
> > > > > > > > > >>>>>>> the 2.x versions.
> > > > > > > > > >>>>>>> Actually I think there are some dependency version
> > > > > resolution
> > > > > > > > > >>>> policies
> > > > > > > > > >>>>>> out
> > > > > > > > > >>>>>>> there which picks the highest minor version when
> the
> > > > > > > dependencies
> > > > > > > > > >>>> pull
> > > > > > > > > >>>>> in
> > > > > > > > > >>>>>>> multiple minor versions of the same jar, which may
> be
> > > > > broken
> > > > > > if
> > > > > > > > > >> we
> > > > > > > > > >>>>> remove
> > > > > > > > > >>>>>>> the API in minor releases.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I see your point, that users would feel surprised if
> > > they
> > > > > find
> > > > > > > > > >> things
> > > > > > > > > >>>> no
> > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> > release.
> > > > > > > However,
> > > > > > > > > >> I'd
> > > > > > > > > >>>>> like
> > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > > similar
> > > > > > > > > >> problem
> > > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > > attention
> > > > > > and
> > > > > > > > > >> make
> > > > > > > > > >>>>> sure
> > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> > longer
> > > > be
> > > > > > > Public
> > > > > > > > > >> in
> > > > > > > > > >>>>> 2.0.
> > > > > > > > > >>>>>> There're various ways to do that, e.g., release
> notes,
> > > > > > warnings
> > > > > > > in
> > > > > > > > > >>>> logs,
> > > > > > > > > >>>>>> etc.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Another possible alternative: whenever there's a
> > > > deprecated
> > > > > > > Public
> > > > > > > > > >> API
> > > > > > > > > >>>>> that
> > > > > > > > > >>>>>> reaches a major version bump before the migration
> > > period,
> > > > > and
> > > > > > we
> > > > > > > > > >> also
> > > > > > > > > >>>>> don't
> > > > > > > > > >>>>>> want to carry it for all the next major release
> > series,
> > > we
> > > > > may
> > > > > > > > > >> consider
> > > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > > version
> > > > > > > after
> > > > > > > > > >> the
> > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > > then
> > > > we
> > > > > > > bump
> > > > > > > > > >> to
> > > > > > > > > >>>> 2.0,
> > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > > provide
> > > > > > > users
> > > > > > > > > >>>> another
> > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while
> satisfying
> > > the
> > > > > > > > > >>>> 2-minor-release
> > > > > > > > > >>>>>> migration period.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I think my major point is, we should not carry APIs
> > > > > deprecated
> > > > > > > in
> > > > > > > > a
> > > > > > > > > >>>>>> previous major version along all the next major
> > version
> > > > > > series.
> > > > > > > > I'd
> > > > > > > > > >>>> like
> > > > > > > > > >>>>> to
> > > > > > > > > >>>>>> try giving users more commitments, i.e. the
> migration
> > > > > period,
> > > > > > as
> > > > > > > > > >> long
> > > > > > > > > >>>> as
> > > > > > > > > >>>>> it
> > > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> > it
> > > > > > doesn't
> > > > > > > > > >> work,
> > > > > > > > > >>>> I'd
> > > > > > > > > >>>>>> be in favor of not providing the migration period,
> but
> > > > > > fallback
> > > > > > > to
> > > > > > > > > >> only
> > > > > > > > > >>>>>> guarantee the compatibility within the major
> version.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Best,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Xintong
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > > > > vvcephei@apache.org
> > > > > > > > > <ma...@apache.org>
> > > > > > > > > >>>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hi Becket,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> > > > conversation
> > > > > > > about
> > > > > > > > > >>>>>>> compatibility outside of this FLIP thread, but for
> > > now, I
> > > > > can
> > > > > > > > > >> accept
> > > > > > > > > >>>>> your
> > > > > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Thanks again,
> > > > > > > > > >>>>>>> John
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > > > >>>>>>>> Hi John,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in
> > major
> > > > > > > releases
> > > > > > > > > >>>>> across
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > > PublicEvolving
> > > > > > APIs
> > > > > > > > > >> are
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> > should
> > > > be
> > > > > > > > > >> reflected
> > > > > > > > > >>>>> in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> > must
> > > > keep
> > > > > > > them
> > > > > > > > > >>>>> around
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we
> can
> > > > drop
> > > > > > them
> > > > > > > > > >> in a
> > > > > > > > > >>>>>>> minor or
> > > > > > > > > >>>>>>>>> patch release.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Personally speaking, I would love to do this, for
> > > > exactly
> > > > > > the
> > > > > > > > > >>>> reason
> > > > > > > > > >>>>>> you
> > > > > > > > > >>>>>>>> mentioned. However, I did not propose this due to
> > the
> > > > > > > following
> > > > > > > > > >>>>>> reasons:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> > > > > accepted
> > > > > > > > > >> FLIPs
> > > > > > > > > >>>> too
> > > > > > > > > >>>>>>> soon.
> > > > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > > > > development.
> > > > > > At
> > > > > > > > > >> this
> > > > > > > > > >>>>>>> point,
> > > > > > > > > >>>>>>>> Flink still lacks some design / routines to
> support
> > > good
> > > > > API
> > > > > > > > > >>>>>>> evolvability /
> > > > > > > > > >>>>>>>> extensibility. Just like you said, it takes some
> > time
> > > to
> > > > > be
> > > > > > > > > >> good at
> > > > > > > > > >>>>>> this.
> > > > > > > > > >>>>>>>> In this case, my concern is that only removing
> > > > > Experimental
> > > > > > /
> > > > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may
> > > result
> > > > in
> > > > > > too
> > > > > > > > > >> much
> > > > > > > > > >>>>>>>> overhead and dramatically slow down the
> development
> > of
> > > > > > Flink.
> > > > > > > > > >> So, I
> > > > > > > > > >>>>> was
> > > > > > > > > >>>>>>>> thinking that we can start with the current
> status.
> > > > > > Hopefully
> > > > > > > > > >> after
> > > > > > > > > >>>>> we
> > > > > > > > > >>>>>>> are
> > > > > > > > > >>>>>>>> more comfortable with the maintenance overhead of
> > > > > deprecated
> > > > > > > > > >> APIs,
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>> can
> > > > > > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > > > > > >> PublicEvolving
> > > > > > > > > >>>>> APIs.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi Becket,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation
> process
> > is
> > > > > > really
> > > > > > > > > >>>>>> important.
> > > > > > > > > >>>>>>> I
> > > > > > > > > >>>>>>>>> understand some people’s concerns about the
> > > additional
> > > > > > burden
> > > > > > > > > >> for
> > > > > > > > > >>>>>>> project
> > > > > > > > > >>>>>>>>> maintainers, but my personal experience with
> Kafka
> > > has
> > > > > been
> > > > > > > > > >> that
> > > > > > > > > >>>>> it’s
> > > > > > > > > >>>>>>> very
> > > > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to
> > > users.
> > > > > In
> > > > > > > > > >> fact,
> > > > > > > > > >>>>> users
> > > > > > > > > >>>>>>>>> being able to confidently upgrade is also a
> benefit
> > > to
> > > > > > > > > >>>> maintainers,
> > > > > > > > > >>>>> as
> > > > > > > > > >>>>>>> we
> > > > > > > > > >>>>>>>>> will get fewer questions from people stuck on
> very
> > > old
> > > > > > > > > >> versions.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> One question:
> > > > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in
> > > major
> > > > > > > > > >> releases
> > > > > > > > > >>>>> across
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > > PublicEvolving
> > > > > > APIs
> > > > > > > > > >> are
> > > > > > > > > >>>> by
> > > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> > should
> > > > be
> > > > > > > > > >> reflected
> > > > > > > > > >>>>> in
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> > must
> > > > keep
> > > > > > > them
> > > > > > > > > >>>>> around
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we
> can
> > > > drop
> > > > > > them
> > > > > > > > > >> in a
> > > > > > > > > >>>>>>> minor or
> > > > > > > > > >>>>>>>>> patch release.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The advantage of forbidding the removal of any
> API
> > in
> > > > > minor
> > > > > > > or
> > > > > > > > > >>>> patch
> > > > > > > > > >>>>>>>>> releases is that users will get a strong
> guarantee
> > > that
> > > > > > they
> > > > > > > > > >> can
> > > > > > > > > >>>>> bump
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>> minor or patch version and still be able to
> > compile,
> > > or
> > > > > > even
> > > > > > > > > >> just
> > > > > > > > > >>>>>>> re-link
> > > > > > > > > >>>>>>>>> and know that they won’t face “MethodDef”
> > exceptions
> > > at
> > > > > run
> > > > > > > > > >> time.
> > > > > > > > > >>>>> This
> > > > > > > > > >>>>>>> is a
> > > > > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> > > > Experimental
> > > > > > > APIs
> > > > > > > > > >>>>> outside
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>>>> major releases, users can no longer confidently
> > > > upgrade.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple
> > of
> > > > > > points:
> > > > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation
> > instead
> > > of
> > > > > > > > > >>>> introducing
> > > > > > > > > >>>>>> our
> > > > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is
> well
> > > > > > > integrated
> > > > > > > > > >>>> into
> > > > > > > > > >>>>>>> build
> > > > > > > > > >>>>>>>>> tools and IDEs.
> > > > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in
> this
> > > > FLIP;
> > > > > > it
> > > > > > > > > >> seems
> > > > > > > > > >>>>>>>>> orthogonal, and something that should probably be
> > > taken
> > > > > > > > > >>>> case-by-case
> > > > > > > > > >>>>>>>>> anyway.
> > > > > > > > > >>>>>>>>> * Aside from deprecation and removal, there have
> > been
> > > > > some
> > > > > > > > > >>>>> discussions
> > > > > > > > > >>>>>>>>> about how to evolve APIs and behavior in
> compatible
> > > > ways.
> > > > > > > > > >> This is
> > > > > > > > > >>>>>>> somewhat
> > > > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it
> > > > before,
> > > > > > > it’ll
> > > > > > > > > >>>> take
> > > > > > > > > >>>>>> some
> > > > > > > > > >>>>>>>>> time to become good at it. I feel like this topic
> > > > should
> > > > > > also
> > > > > > > > > >> be
> > > > > > > > > >>>>>>> orthogonal
> > > > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to
> > > > adopt a
> > > > > > > > > >> simple
> > > > > > > > > >>>>>> policy
> > > > > > > > > >>>>>>> not
> > > > > > > > > >>>>>>>>> to break existing user programs, and leave the
> > “how”
> > > up
> > > > > to
> > > > > > > > > >>>>>> implementers
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>>> reviewers.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > > >>>>>>>>> John
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > > > >>>>>>>>>> Hi All,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by
> Xintong
> > > is
> > > > a
> > > > > > > > > >> great
> > > > > > > > > >>>>> idea.
> > > > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired,
> i.e.
> > > > > > > > > >>>> @PublicEvolving
> > > > > > > > > >>>>>> --(2
> > > > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1
> > major
> > > > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a
> lot
> > of
> > > > > > > > > >> flexibility
> > > > > > > > > >>>>>>> without
> > > > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are
> > allowed
> > > to
> > > > > > > > > >> change
> > > > > > > > > >>>>>> between
> > > > > > > > > >>>>>>>>> major
> > > > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> > > > > provides
> > > > > > > > > >>>>> additional
> > > > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> > > > > themself
> > > > > > > > > >> are
> > > > > > > > > >>>> not
> > > > > > > > > >>>>>>>>> changed.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> > > > issues. I
> > > > > > > > > >> want to
> > > > > > > > > >>>>>> move
> > > > > > > > > >>>>>>> one
> > > > > > > > > >>>>>>>>>> step further and suggest introducing one more
> > > > annotation
> > > > > > > > > >>>> @Retired.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a
> compromise
> > of
> > > > > > > > > >> downgrading
> > > > > > > > > >>>>>>> @Public
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my
> > reply,
> > > > > Java
> > > > > > > > > >>>> standard
> > > > > > > > > >>>>>>>>>> @deprecated should be used in the early stage of
> > the
> > > > > > > > > >> deprecation
> > > > > > > > > >>>>>>> process
> > > > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since
> > Java
> > > > does
> > > > > > not
> > > > > > > > > >>>> allow
> > > > > > > > > >>>>>> us
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>> extend annotation, I think it would be feasible
> to
> > > > have
> > > > > > the
> > > > > > > > > >> new
> > > > > > > > > >>>>>>> @Retired
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>> help us monitor and manage the deprecation
> > process,
> > > > > house
> > > > > > > > > >>>>> cleaning,
> > > > > > > > > >>>>>>> etc.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> @Retired:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> 1. There must be a replacement with
> functionality
> > > > > > > > > >> compatibility
> > > > > > > > > >>>>>> before
> > > > > > > > > >>>>>>>>> APIs
> > > > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> > > > > processes
> > > > > > > > > >> on
> > > > > > > > > >>>> the
> > > > > > > > > >>>>> ML
> > > > > > > > > >>>>>>> are
> > > > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed
> after 1
> > > > minor
> > > > > > > > > >> release
> > > > > > > > > >>>>>>> sharply
> > > > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check
> > > > whether
> > > > > > it
> > > > > > > > > >> is
> > > > > > > > > >>>>>>> possible).
> > > > > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible
> > to
> > > > > remove
> > > > > > > > > >> them.
> > > > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving
> ->
> > > > > > @Retired
> > > > > > > > > >> are
> > > > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and
> > @Internal
> > > > ->
> > > > > > > > > >> @Retired
> > > > > > > > > >>>>>> could
> > > > > > > > > >>>>>>>>> also
> > > > > > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > > > > > >>>> dev-friendliness,
> > > > > > > > > >>>>>> but
> > > > > > > > > >>>>>>> not
> > > > > > > > > >>>>>>>>>> mandatory.
> > > > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to
> > > > support
> > > > > > the
> > > > > > > > > >>>>>>> deprecation
> > > > > > > > > >>>>>>>>>> process management. Further extension is
> possible,
> > > > since
> > > > > > the
> > > > > > > > > >>>>>>> annotation
> > > > > > > > > >>>>>>>>> is
> > > > > > > > > >>>>>>>>>> built by us.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>> Jing
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the
> > replies
> > > > > inline
> > > > > > > > > >>>> below.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a
> > > Public
> > > > > API
> > > > > > > > > >> is
> > > > > > > > > >>>>> also a
> > > > > > > > > >>>>>>>>> kind of
> > > > > > > > > >>>>>>>>>>>> such change, just like removing one, which can
> > > only
> > > > > > > > > >> happen
> > > > > > > > > >>>> with
> > > > > > > > > >>>>>>> major
> > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> > demoting
> > > > > > Public
> > > > > > > > > >>>> APIs
> > > > > > > > > >>>>>>>>> anytime,
> > > > > > > > > >>>>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> > before
> > > > > > > > > >> reaching
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > > demoting
> > > > > > > > > >> would
> > > > > > > > > >>>> be a
> > > > > > > > > >>>>>>> weaker
> > > > > > > > > >>>>>>>>>>>> change compared to removing the API
> immediately
> > > upon
> > > > > > > > > >> major
> > > > > > > > > >>>>>> version
> > > > > > > > > >>>>>>>>> bumps,
> > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > > 2-minor-release
> > > > > > > > > >>>>>> migration
> > > > > > > > > >>>>>>>>>>> period.
> > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > > `@PublicEvolving`
> > > > > > > > > >> sounds
> > > > > > > > > >>>>>>> against
> > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> > if
> > > > > > > > > >> necessary,
> > > > > > > > > >>>>>> e.g.,
> > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I
> > > don't
> > > > > > > > > >> change my
> > > > > > > > > >>>>> code
> > > > > > > > > >>>>>>> at
> > > > > > > > > >>>>>>>>> all,
> > > > > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > > > issue, I
> > > > > > am
> > > > > > > > > >> fine
> > > > > > > > > >>>>>> with
> > > > > > > > > >>>>>>> all
> > > > > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > > > > >>>>>>>>>>> Actually I think there are some dependency
> > version
> > > > > > > > > >> resolution
> > > > > > > > > >>>>>>> policies
> > > > > > > > > >>>>>>>>> out
> > > > > > > > > >>>>>>>>>>> there which picks the highest minor version
> when
> > > the
> > > > > > > > > >>>> dependencies
> > > > > > > > > >>>>>>> pull
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which
> > may
> > > be
> > > > > > > > > >> broken if
> > > > > > > > > >>>>> we
> > > > > > > > > >>>>>>>>> remove
> > > > > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > > "legal"
> > > > > > that
> > > > > > > > > >> we
> > > > > > > > > >>>>> bump
> > > > > > > > > >>>>>> up
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > > needed.
> > > > > > > > > >> However,
> > > > > > > > > >>>>> this
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that
> public
> > > APIs
> > > > > > will
> > > > > > > > > >>>> stay
> > > > > > > > > >>>>>>> stable
> > > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> > can
> > > > end
> > > > > > > > > >>>> anytime.
> > > > > > > > > >>>>>>> IMHO,
> > > > > > > > > >>>>>>>>>>> short
> > > > > > > > > >>>>>>>>>>>> major release series are not something "make
> the
> > > end
> > > > > > > > > >> users
> > > > > > > > > >>>>>> happy",
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> > make
> > > > > > > > > >> frequent
> > > > > > > > > >>>>>> breaking
> > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > > approach,
> > > > > we
> > > > > > > > > >> can
> > > > > > > > > >>>>>> still
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> longer major release series, while only
> allowing
> > > > > Public
> > > > > > > > > >> APIs
> > > > > > > > > >>>>>>>>> deprecated
> > > > > > > > > >>>>>>>>>>> at
> > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > > removed
> > > > in
> > > > > > > > > >> the
> > > > > > > > > >>>> next
> > > > > > > > > >>>>>>> major
> > > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I totally agree that frequent major version
> bumps
> > > are
> > > > > not
> > > > > > > > > >>>> ideal,
> > > > > > > > > >>>>>> but
> > > > > > > > > >>>>>>>>> here
> > > > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump
> > which
> > > > > > > > > >> removes a
> > > > > > > > > >>>>>> Public
> > > > > > > > > >>>>>>>>> API.
> > > > > > > > > >>>>>>>>>>> So the context is that we have already decided
> to
> > > > > remove
> > > > > > > > > >> this
> > > > > > > > > >>>>>> Public
> > > > > > > > > >>>>>>> API
> > > > > > > > > >>>>>>>>>>> while keeping everything else backwards
> > > compatible. I
> > > > > > > > > >> think a
> > > > > > > > > >>>>> major
> > > > > > > > > >>>>>>>>> version
> > > > > > > > > >>>>>>>>>>> bump is a commonly understood signal here,
> > compared
> > > > > with
> > > > > > a
> > > > > > > > > >>>> minor
> > > > > > > > > >>>>>>> version
> > > > > > > > > >>>>>>>>>>> change. From end users' perspective, for those
> > who
> > > > are
> > > > > > not
> > > > > > > > > >>>>>> impacted,
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>> this case upgrading a major version is not
> > > > necessarily
> > > > > > more
> > > > > > > > > >>>>>> involved
> > > > > > > > > >>>>>>>>> than
> > > > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as
> > > smooth
> > > > > as a
> > > > > > > > > >>>>>> dependency
> > > > > > > > > >>>>>>>>>>> version change. For those who are impacted,
> they
> > > will
> > > > > > lose
> > > > > > > > > >> the
> > > > > > > > > >>>>>> Public
> > > > > > > > > >>>>>>>>> API
> > > > > > > > > >>>>>>>>>>> anyways and a major version bump ensures there
> is
> > > no
> > > > > > > > > >> surprise.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > > > > >>>>>> tonysong820@gmail.com <mailto:tonysong820@gmail.com
> >
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Public API is a well defined common concept,
> and
> > > one
> > > > > of
> > > > > > > > > >> its
> > > > > > > > > >>>>>>>>>>>>> convention is that it only changes with a
> major
> > > > > version
> > > > > > > > > >>>>> change.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a
> > > > Public
> > > > > > > > > >> API is
> > > > > > > > > >>>>>> also a
> > > > > > > > > >>>>>>>>> kind
> > > > > > > > > >>>>>>>>>>>> of such change, just like removing one, which
> > can
> > > > only
> > > > > > > > > >> happen
> > > > > > > > > >>>>>> with
> > > > > > > > > >>>>>>>>> major
> > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> > demoting
> > > > > > Public
> > > > > > > > > >>>> APIs
> > > > > > > > > >>>>>>>>> anytime,
> > > > > > > > > >>>>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> > before
> > > > > > > > > >> reaching
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > > demoting
> > > > > > > > > >> would
> > > > > > > > > >>>> be a
> > > > > > > > > >>>>>>> weaker
> > > > > > > > > >>>>>>>>>>>> change compared to removing the API
> immediately
> > > upon
> > > > > > > > > >> major
> > > > > > > > > >>>>>> version
> > > > > > > > > >>>>>>>>> bumps,
> > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > > 2-minor-release
> > > > > > > > > >>>>>> migration
> > > > > > > > > >>>>>>>>>>> period.
> > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > > `@PublicEvolving`
> > > > > > > > > >> sounds
> > > > > > > > > >>>>>>> against
> > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> > if
> > > > > > > > > >> necessary,
> > > > > > > > > >>>>>> e.g.,
> > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> But it should be
> > > > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if
> > we
> > > > > really
> > > > > > > > > >> want
> > > > > > > > > >>>>> to
> > > > > > > > > >>>>>>> get
> > > > > > > > > >>>>>>>>> rid
> > > > > > > > > >>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > > > "legal"
> > > > > > > > > >> that we
> > > > > > > > > >>>>>> bump
> > > > > > > > > >>>>>>> up
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > > needed.
> > > > > > > > > >> However,
> > > > > > > > > >>>>> this
> > > > > > > > > >>>>>>> also
> > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that
> public
> > > APIs
> > > > > > will
> > > > > > > > > >>>> stay
> > > > > > > > > >>>>>>> stable
> > > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> > can
> > > > end
> > > > > > > > > >>>> anytime.
> > > > > > > > > >>>>>>> IMHO,
> > > > > > > > > >>>>>>>>>>> short
> > > > > > > > > >>>>>>>>>>>> major release series are not something "make
> the
> > > end
> > > > > > > > > >> users
> > > > > > > > > >>>>>> happy",
> > > > > > > > > >>>>>>> but
> > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> > make
> > > > > > > > > >> frequent
> > > > > > > > > >>>>>> breaking
> > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > > approach,
> > > > > we
> > > > > > > > > >> can
> > > > > > > > > >>>>>> still
> > > > > > > > > >>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> longer major release series, while only
> allowing
> > > > > Public
> > > > > > > > > >> APIs
> > > > > > > > > >>>>>>>>> deprecated
> > > > > > > > > >>>>>>>>>>> at
> > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > > removed
> > > > in
> > > > > > > > > >> the
> > > > > > > > > >>>> next
> > > > > > > > > >>>>>>> major
> > > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Given our track record I would prefer a
> regular
> > > > cycle
> > > > > > > > > >> (1-2
> > > > > > > > > >>>>> years)
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> > not
> > > > put
> > > > > > it
> > > > > > > > > >>>> again
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > > expectation
> > > > > > on
> > > > > > > > > >>>> when
> > > > > > > > > >>>>>>>>> breaking
> > > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a
> good
> > > > time
> > > > > > > > > >> for new
> > > > > > > > > >>>>>> major
> > > > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking
> > changes
> > > > > > > > > >> needed.
> > > > > > > > > >>>> That
> > > > > > > > > >>>>>>> makes
> > > > > > > > > >>>>>>>>> 1-2
> > > > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic,
> while
> > > > > leaving
> > > > > > > > > >> us
> > > > > > > > > >>>> more
> > > > > > > > > >>>>>>> time
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Xintong
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay
> > Schepler
> > > <
> > > > > > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid
> releasing
> > > > minor
> > > > > > > > > >>>>> versions
> > > > > > > > > >>>>>>> for
> > > > > > > > > >>>>>>>>>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not
> sure
> > > how
> > > > > > > > > >> viable
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>> is
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> practice.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially
> > dropping
> > > > > > > > > >> support
> > > > > > > > > >>>> for
> > > > > > > > > >>>>>> Java
> > > > > > > > > >>>>>>>>> 8/11,
> > > > > > > > > >>>>>>>>>>>>> which may very well be a problem for our
> > current
> > > > > users.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> > > > confusion
> > > > > > > > > >> about
> > > > > > > > > >>>>>> Public
> > > > > > > > > >>>>>>>>> API
> > > > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a
> > > > mistake
> > > > > > > > > >> in
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> NOTES
> > > > > > > > > >>>>>>>>> part
> > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I
> just
> > > > fixed
> > > > > > > > > >> it.
> > > > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear
> > that
> > > > we
> > > > > > > > > >> are on
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>> same
> > > > > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version
> if
> > we
> > > > > > > > > >> really
> > > > > > > > > >>>> want
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> get
> > > > > > > > > >>>>>>>>>>> rid
> > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took
> > to
> > > > get
> > > > > us
> > > > > > > > > >> to
> > > > > > > > > >>>>> 2.0.
> > > > > > > > > >>>>>> ;)
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on
> the
> > > > > cadence
> > > > > > > > > >> of
> > > > > > > > > >>>>> major
> > > > > > > > > >>>>>>>>> releases
> > > > > > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> > > > > opinions
> > > > > > > > > >> on
> > > > > > > > > >>>>> that.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor
> > releases
> > > > > > between
> > > > > > > > > >>>> major
> > > > > > > > > >>>>>>> ones,
> > > > > > > > > >>>>>>>>>>>>> that for us means that users have to
> > potentially
> > > > deal
> > > > > > > > > >> with
> > > > > > > > > >>>>>>> breaking
> > > > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a
> lot.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a
> regular
> > > > cycle
> > > > > > > > > >> (1-2
> > > > > > > > > >>>>>> years)
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> > not
> > > > put
> > > > > > it
> > > > > > > > > >>>> again
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > > expectation
> > > > > > on
> > > > > > > > > >>>> when
> > > > > > > > > >>>>>>>>> breaking
> > > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate
> > > > thread.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change
> in
> > > > mind?
> > > > > > > > > >> Not
> > > > > > > > > >>>>> sure
> > > > > > > > > >>>>>> I
> > > > > > > > > >>>>>>>>> fully
> > > > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change
> > > here.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> > > > performance
> > > > > in
> > > > > > > > > >>>>> certain
> > > > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> > > > unknowingly)
> > > > > > > > > >> relied
> > > > > > > > > >>>>> upon
> > > > > > > > > >>>>>>>>>>>>> (
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > > > > ),
> > > > > > > > > a semantic change to some
> > > > > > > > > >> API.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> > > > submission.
> > > > > A
> > > > > > > > > >> few
> > > > > > > > > >>>>>>> releases
> > > > > > > > > >>>>>>>>> back
> > > > > > > > > >>>>>>>>>>>>> we made changes such that the initialization
> of
> > > the
> > > > > job
> > > > > > > > > >>>> master
> > > > > > > > > >>>>>>>>> happens
> > > > > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > > > > >>>>>>>>>>>>> This meant the job submission call returns
> > > sooner,
> > > > > and
> > > > > > > > > >> the
> > > > > > > > > >>>> job
> > > > > > > > > >>>>>>> state
> > > > > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible
> change,
> > > but
> > > > > the
> > > > > > > > > >>>>> observed
> > > > > > > > > >>>>>>>>> behavior
> > > > > > > > > >>>>>>>>>>>>> may be different.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over
> > time
> > > we
> > > > > > > > > >> changed
> > > > > > > > > >>>>> what
> > > > > > > > > >>>>>>> some
> > > > > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong, Jark and Jing,

Thanks for the reply. Yes, we can only mark the DataStream API as
@Deprecated after the ProcessFunction API is fully functional and mature.

It is a fair point that the condition of marking a @Public API as
deprecated should also be a part of this FLIP. I just added that to the
FLIP wiki. This is probably more of a clarification on the existing
convention, rather than a change.

It looks like we are on the same page now for this FLIP. If so, I'll start
a VOTE thread in two days.

Thanks,

Jiangjie (Becket) Qin

On Mon, Jun 26, 2023 at 8:09 PM Xintong Song <to...@gmail.com> wrote:

> >
> > Considering DataStream API is the most fundamental and complex API of
> > Flink, I think it is worth a longer time than the general process for the
> > deprecation period to wait for the new API be mature.
> >
>
> This inspires me. In this specific case, compared to how long should
> DataStream API be removed after deprecation, it's probably more important
> to answer the question how long would ProcessFunction API become mature and
> stable after being introduced. According to FLIP-197[1], it requires 4
> minor releases by default to promote an @Experimental API to @Public. And
> for ProcessFunction API, which aims to replace DataStream API as one of the
> most fundamental API of Flink, I'd expect this to take at least the default
> time, or even longer. And we probably should wait until we believe
> ProcessFunction API is stable to mark DataStream API as deprecated, rather
> than as soon as it's introduced. Assuming we introduce the ProcessFunction
> API in 2.0, that means we would need to wait for 6 minor releases (4 for
> the new API to become stable, and 2 for the migration period) to remove
> DataStream API, which is ~2.5 year (assuming 5 months / minor release),
> which sounds acceptable for another major version bump.
>
> To wrap things up, it seems to me, sadly, that anyway we cannot avoid the
> overhead for maintaining both DataStream & ProcessFunction APIs for at
> least 6 minor releases.
>
> Best,
>
> Xintong
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
>
>
>
> On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi all,
> >
> > Just want to make sure we are on the same page. There is another
> example[1]
> > I was aware of recently that shows why more factors need to be taken care
> > of than just the migration period. Thanks Galen for your hint.
> >
> > To put it simply, the concern about API deprecation is not that
> deprecated
> > APIs have been removed too early (min migration period is required). The
> > major concern is that APIs are marked as deprecated for a (too) long
> time,
> > much longer than the migration period discussed in this thread, afaik.
> > Since there is no clear picture/definition, no one knows when to do the
> > migration for users(after the migration period has expired) and when to
> > remove deprecated APIs for Flink developers.
> >
> > Based on all the information I knew, there are two kinds of obstacles
> that
> > will and should block the deprecation process:
> >
> > 1. Lack of functionalities in new APIs. It happens e.g. with the
> > SourceFunction to FLIP-27 Source migration. Users who rely on those
> > functions can not migrate to new APIs.
> > 2. new APIs have critical bugs. An example could be found at [1]. Users
> > have to stick to the deprecated APIs.
> >
> > Since FLIP-321 is focusing on the API deprecation process, those blocking
> > issues deserve attention and should be put into the FLIP. The current
> FLIP
> > seems to only focus on migration periods. If we consider those blocking
> > issues as orthogonal issues that are beyond the scope of this discussion,
> > does it make sense to change the FLIP title to something like "Introduce
> > minimum migration periods of API deprecation process"?
> >
> > Best regards,
> > Jing
> >
> > [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
> >
> > On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > I agree with Jingsong and Becket.
> > >
> > > Look at the legacy SourceFunction (a small part of DataStream API),
> > > the SourceFunction is still not and can't be marked deprecated[1] until
> > > now after the new Source was released 2 years ago, because the new
> Source
> > > still can't fully consume the abilities of legacy API. Considering
> > > DataStream
> > > API is the most fundamental and complex API of Flink, I think it is
> worth
> > > a longer time than the general process for the deprecation period to
> > > wait for the new API be mature. The above 2 options sound a bit of rush
> > > for such a widely used API.
> > >
> > > I fully understand the concern of maintenance overhead, but it's a bit
> > hard
> > > for others to estimate maintenance costs without a concrete design and
> > code
> > > of the new ProcessFunction API. I agree with Becket that maybe we can
> > > re-evaluate the API deprecation process once we have the new
> > > ProcessFunction
> > > API. If the maintenance is indeed huge, I think it is reasonable to
> have
> > a
> > > special rule for this case at that time.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]: https://issues.apache.org/jira/browse/FLINK-28045
> > >
> > > On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com> wrote:
> > >
> > > > Hi Jingsong,
> > > >
> > > > Thanks for the reply. I completely agree with you.
> > > >
> > > > The above 2 options are based on the assumption that the community
> > cannot
> > > > afford to maintain the deprecated DataStream API for long. I'd say we
> > > > should try everything we can to maintain it for as much time as
> > possible.
> > > > DataStream API is actually the most used API in Flink by so many
> users
> > at
> > > > this point. Removing it any time soon will dramatically hurt our
> users.
> > > So
> > > > ideally we should keep it for at least 2 years after deprecation, if
> > not
> > > > more.
> > > >
> > > > The prohibitively high maintenance overhead is just an assumption.
> > > > Personally speaking, I don't feel this assumption is necessarily
> true.
> > We
> > > > should re-evaluate once we have the new ProcessFunction API in place.
> > > > Without the code it is hard to tell for sure. I am actually kind of
> > > > optimistic about the maintenance cost.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <jingsonglee0@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks Becket and all for your discussion.
> > > > >
> > > > > > 1. We say this FLIP is enforced starting release 2.0. For current
> > 1.x
> > > > > APIs,
> > > > > we provide a migration period with best effort, while allowing
> > > exceptions
> > > > > for immediate removal in 2.0. That means we will still try with
> best
> > > > effort
> > > > > to get the ProcessFuncion API ready and deprecate the DataStream
> API
> > in
> > > > > 1.x, but will also be allowed to remove DataStream API in 2.0 if
> it's
> > > not
> > > > > deprecated 2 minor releases before the major version bump.
> > > > >
> > > > > > 2. We strictly follow the process in this FLIP, and will quickly
> > bump
> > > > the
> > > > > major version from 2.x to 3.0 once the migration period for
> > DataStream
> > > > API
> > > > > is reached.
> > > > >
> > > > > Sorry, I didn't read the previous detailed discussion because the
> > > > > discussion list was so long.
> > > > >
> > > > > I don't really like either of these options.
> > > > >
> > > > > Considering that DataStream is such an important API, can we offer
> a
> > > > third
> > > > > option:
> > > > >
> > > > > 3. Maintain the DataStream API throughout 2.X and remove it until
> > 3.x.
> > > > But
> > > > > there's no need to assume that 2.X is a short version, it's still a
> > > > normal
> > > > > major version.
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> > > > >
> > > > > > Thanks much for the input, John, Stefan and Jing.
> > > > > >
> > > > > > I think Xingtong has well summarized the pros and cons of the two
> > > > > options.
> > > > > > Let's collect a few more opinions here and we can move forward
> with
> > > the
> > > > > one
> > > > > > more people prefer.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thanks Xingtong for the summary. If I could only choose one of
> > the
> > > > > given
> > > > > > > two options, I would go with option 1. I understood that
> option 2
> > > > > worked
> > > > > > > great with Kafka. But the bridge release will still confuse
> users
> > > and
> > > > > my
> > > > > > > gut feeling is that many users will skip 2.0 and be waiting for
> > 3.0
> > > > or
> > > > > > even
> > > > > > > 3.x. And since fewer users will use Flink 2.x, the development
> > > focus
> > > > > will
> > > > > > > be on Flink 3.0 with the fact that the current Flink release is
> > > 1.17
> > > > > and
> > > > > > we
> > > > > > > are preparing 2.0 release. That is weird for me.
> > > > > > >
> > > > > > > THB, I would not name the change from @Public to @Retired as a
> > > > > demotion.
> > > > > > > The purpose of @Retire is to extend the API lifecycle with one
> > more
> > > > > > stage,
> > > > > > > like in the real world, people born, studied, graduated,
> worked,
> > > and
> > > > > > > retired. Afaiu from the previous discussion, there are two
> rules
> > > we'd
> > > > > > like
> > > > > > > to follow simultaneously:
> > > > > > >
> > > > > > > 1. Public APIs can only be changed between major releases.
> > > > > > > 2. A smooth migration phase should be offered to users, i.e. at
> > > > least 2
> > > > > > > minor releases after APIs are marked as @deprecated. There
> should
> > > be
> > > > > new
> > > > > > > APIs as the replacement.
> > > > > > >
> > > > > > > Agree, those rules are good to improve the user friendliness.
> > > Issues
> > > > we
> > > > > > > discussed are rising because we want to fulfill both of them.
> If
> > we
> > > > > take
> > > > > > > care of deprecation very seriously, APIs can be marked as
> > > > @Deprecated,
> > > > > > only
> > > > > > > when the new APIs as the replacement provide all
> functionalities
> > > the
> > > > > > > deprecated APIs have. In an ideal case without critical bugs
> that
> > > > might
> > > > > > > stop users adopting the new APIs. Otherwise the expected
> > > > "replacement"
> > > > > > will
> > > > > > > not happen. Users will still stick to the deprecated APIs,
> > because
> > > > the
> > > > > > new
> > > > > > > APIs can not be used. For big features, it will need at least 4
> > > minor
> > > > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > > > > >
> > > > > > > - 1st minor release to build the new APIs as the replacement
> and
> > > > > waiting
> > > > > > > for feedback. It might be difficult to mark the old API as
> > > deprecated
> > > > > in
> > > > > > > this release, because we are not sure if the new APIs could
> cover
> > > > 100%
> > > > > > > functionalities.
> > > > > > > -  In the lucky case,  mark all old APIs as deprecated in the
> 2nd
> > > > minor
> > > > > > > release. (I would even suggest having the new APIs released at
> > > least
> > > > > for
> > > > > > > two minor releases before marking it as deprecated to make sure
> > > they
> > > > > can
> > > > > > > really replace the old APIs, in case we care more about smooth
> > > > > migration)
> > > > > > > - 3rd minor release for the migration period
> > > > > > > -  In another lucky case, the 4th release is a major release,
> the
> > > > > > > deprecated APIs could be removed.
> > > > > > >
> > > > > > > The above described scenario works only in an ideal case. In
> > > reality,
> > > > > it
> > > > > > > might take longer to get the new APIs ready and mark the old
> API
> > > > > > > deprecated. Furthermore, if the 4th release is not a major
> > release,
> > > > we
> > > > > > will
> > > > > > > have to maintain both APIs for many further minor releases. The
> > > > > question
> > > > > > is
> > > > > > > how to know the next major release in advance, especially 4
> minor
> > > > > > releases'
> > > > > > > period, i.e. more than 2 years in advance? Given that Flink
> > > contains
> > > > > many
> > > > > > > modules, it is difficult to ask devs to create a 2-3 years
> > > > deprecation
> > > > > > plan
> > > > > > > for each case. In case we want to build major releases at a
> fast
> > > > pace,
> > > > > > > let's say every two years, it means devs must plan any API
> > > > deprecation
> > > > > > > right after each major release. Afaiac, it is quite difficult.
> > > > > > >
> > > > > > > The major issue is, afaiu, if we follow rule 2, we have to keep
> > all
> > > > > > @Public
> > > > > > > APIs, e.g. DataStream, that are not marked as deprecated yet,
> to
> > > 2.0.
> > > > > > Then
> > > > > > > we have to follow rule 1 to keep it unchanged until we have
> 3.0.
> > > That
> > > > > is
> > > > > > > why @Retired is useful to give devs more flexibility and still
> > > > fulfill
> > > > > > both
> > > > > > > rules. Let's check it with examples:
> > > > > > >
> > > > > > > - we have @Public DataStream API in 1.18. It will not be marked
> > > > > > > as @Deprecated, because the new APIs as the replacement are not
> > > > ready.
> > > > > > > - we keep the DataStream API itself unchanged in 2.0, but
> change
> > > the
> > > > > > > annotation from @Public to @Retire. New APIs will be introduced
> > > too.
> > > > In
> > > > > > > this case, Rule 1 is ok, since public API is allowed to change
> > > > between
> > > > > > > major releases. Only changing annotation is the minimal change
> we
> > > > could
> > > > > > do
> > > > > > > and it does not break rule 1. Rule 2 is ok too, since the
> > > DataStream
> > > > > APIs
> > > > > > > work exactly the same.  Attention: the change of @Public ->
> > > @Retired
> > > > > can
> > > > > > > only be done between major releases, because @Public APIs can
> > only
> > > be
> > > > > > > changed between major releases.
> > > > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > > > - in 2.3, DataStream will be removed.
> > > > > > >
> > > > > > > Becket mentioned previously (please correct me if I didn't
> > > understand
> > > > > it
> > > > > > > correctly) that users might not check the changes of annotation
> > and
> > > > the
> > > > > > > upgrade process might not be smooth. I was wondering, if users
> > > don't
> > > > > pay
> > > > > > > attention to that, they will also not pay attention to
> > @deprecated.
> > > > The
> > > > > > > migration will not be smooth too even when we stick to the Java
> > > > > > > standard @deprecated. Let's go throw it with examples:
> > > > > > >
> > > > > > > Scenario 1: with @Retired
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing
> should
> > > be
> > > > > done
> > > > > > > for users who still want to use DataStream. The migration is
> > > smooth.
> > > > > But
> > > > > > > they should be aware of the annotation changes, i.e. @Public ->
> > > > > @Retired
> > > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed.
> > > Since
> > > > > it
> > > > > > is
> > > > > > > a major release upgrade, breaking changes are expected.
> > > > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed.
> > > Since
> > > > > > APIs
> > > > > > > are marked as @Retired, breaking changes are expected too.
> > > > > > >
> > > > > > > In addition to fulfilling both rules, there are two more
> benefits
> > > of
> > > > > this
> > > > > > > option:
> > > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > > > > 2. no bridge release is required. DataStream API could be
> > > deprecated
> > > > in
> > > > > > 2.x
> > > > > > > without breaking any of the two rules. @Retired provides enough
> > > > > > flexibility
> > > > > > > for future deprecations and upgrades.
> > > > > > >
> > > > > > > The only issue I can find is that users might be surprised in
> > case
> > > 3,
> > > > > if
> > > > > > > they ignore the previous @Public -> @Retired change. But for
> such
> > > > > users,
> > > > > > > they will ignore @deprecated annotation too and will have
> issues
> > > with
> > > > > > > pure @deprecated
> > > > > > > annotation too:
> > > > > > >
> > > > > > > Scenario 2: with @deprecated
> > > > > > >
> > > > > > > The example will be slightly different from scenario 1, because
> > > > @Public
> > > > > > > APIs can only be removed between major releases. As an example,
> > > > > > DataStream
> > > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and
> then
> > > > > removed
> > > > > > in
> > > > > > > 2.0
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not
> marked
> > > as
> > > > > > > deprecated are removed between major releases. Rule 1 is ok.
> The
> > 2
> > > > > minor
> > > > > > > releases period defined in rule 2 sounds like a smooth
> migration
> > > > plan,
> > > > > > but
> > > > > > > for users in this case, it is still not smooth.
> > > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade
> process
> > > > with
> > > > > > > deprecated APIs. But for those users who ignore annotation
> > changes,
> > > > > will
> > > > > > > have the same issue mentioned in case 1.
> > > > > > >
> > > > > > > I understood that, compared to scenario 1, users, who ignore
> > > @Public
> > > > > > > -> @Retired
> > > > > > > changes, will not face any breaking changes, after they
> migrated
> > to
> > > > > 2.0.
> > > > > > > But the cost of it is bridge releases. There might be many
> bridge
> > > > > > releases
> > > > > > > in the future. As I mentioned previously, it is not easy to
> find
> > > the
> > > > > > right
> > > > > > > timing to kick off the deprecation process of APIs before the
> > next
> > > > > > expected
> > > > > > > major release and 2-3 years in advance.
> > > > > > >
> > > > > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > > > > @deprecated
> > > > > > > annotation changes
> > > > > > >
> > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed
> in
> > > > 1.21
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no
> > > smooth
> > > > > > > migration.
> > > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > > > > > since @deprecated is ignored
> > > > > > >
> > > > > > > In summary, for users who care about the annotation change,
> they
> > > will
> > > > > get
> > > > > > > more benefits with @Retired. For users who ignore annotation
> > > changes,
> > > > > > they
> > > > > > > will be always facing non-smooth upgrades even using pure
> > standard
> > > > > > > @deprecated
> > > > > > > annotation/process.
> > > > > > >
> > > > > > > For the case that IDEs support @deprecated, that is true. There
> > are
> > > > no
> > > > > > such
> > > > > > > supports for homebuilt annotation. But the intention is to let
> > > users
> > > > be
> > > > > > > aware of it. There will be other ways to do it for users who
> care
> > > > about
> > > > > > > annotation changes. For users who do not care, there should be
> no
> > > > > > > difference between with tool support or without. Using @Retired
> > > with
> > > > > > > @deprecation
> > > > > > > together is at least not worse than only using @deprecation
> > alone.
> > > > And
> > > > > > > there are additional benefits with @Retired.
> > > > > > >
> > > > > > > Just my 2 cents and looking forward to your feedback,
> especially
> > > > > > different
> > > > > > > opinions that will help me understand the issue better. Thanks!
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > > > <sr...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > > Hi Xintong,
> > > > > > > >
> > > > > > > > Thanks for the summary, most of the points that you agreed
> upon
> > > > also
> > > > > > make
> > > > > > > > sense to me.
> > > > > > > >
> > > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > > demoting
> > > > > > > APIs
> > > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> > any
> > > > > > version
> > > > > > > > bump,
> > > > > > > > >   are not good practices.
> > > > > > > >
> > > > > > > > I hope we can can go beyond calling it “not a good practice"
> > and
> > > > > reach
> > > > > > > > consensus that we will not demote or remove public APIs in
> > minor
> > > > > > releases
> > > > > > > > and (technically) enforce this rule for all contributions. I
> > > think
> > > > > > it’s a
> > > > > > > > reasonable expectation for stability from a project as mature
> > as
> > > > > Flink.
> > > > > > > >
> > > > > > > > > I'm personally in favor of option 1. As the
> migration-period
> > > rule
> > > > > is
> > > > > > > > newly
> > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > where
> > > we
> > > > > > > already
> > > > > > > > > missed the best chance for planning the deprecation.
> > Moreover,
> > > I
> > > > do
> > > > > > > > believe
> > > > > > > > > having a quick major version bump does confuse users. Yes,
> we
> > > can
> > > > > > > explain
> > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> anything
> > > > other
> > > > > > than
> > > > > > > > the
> > > > > > > > > removal of an deprecated API. But having to explain this
> > itself
> > > > is
> > > > > an
> > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > understanding,
> > > > > > his
> > > > > > > > > major point is that a quick major version bump causes
> barely
> > > any
> > > > > > actual
> > > > > > > > > lose on users, while in option 1 not providing the
> migration
> > > > period
> > > > > > or
> > > > > > > > > providing a shorter on is an actual lose on users.
> (@Becket,
> > > > please
> > > > > > > > correct
> > > > > > > > > me if I'm mistaken.)
> > > > > > > > >
> > > > > > > >
> > > > > > > > I would be open to both options and maybe postpone this
> > decision
> > > > > until
> > > > > > we
> > > > > > > > have the complete roadmap for Flink 2.0. We would have a
> better
> > > > > > overview
> > > > > > > > about the technical consequences, e.g. how hard it would be
> to
> > > > offer
> > > > > > and
> > > > > > > > maintain both APIs side by side, how well we get both APIs
> > > > separated,
> > > > > > and
> > > > > > > > whether or not we will be able to use the full potential of
> our
> > > > > > breaking
> > > > > > > > changes before the DataStream API is completely removed. One
> > > > example
> > > > > > for
> > > > > > > > the last point I can think of would be moving from lazy to
> > eager
> > > > > state
> > > > > > > > declaration. If we believe that it is reasonable to support
> all
> > > > > planned
> > > > > > > > changes while maintaining both APIs, I'm leaning towards
> option
> > > 2.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stefan
> > > > > > > >
> > > > > > > >
> > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <
> > tonysong820@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Becket and I had an offline voice call earlier today. We
> have
> > > > > reached
> > > > > > > > > consensus on some of the arguments, but still hold
> different
> > > > > opinions
> > > > > > > on
> > > > > > > > > some others. I'd like to post the outcome here for
> > > transparency.
> > > > > > > > >
> > > > > > > > > We both agree that:
> > > > > > > > >
> > > > > > > > >   1. Providing a migration period would be beneficial for
> our
> > > > > users,
> > > > > > > and
> > > > > > > > >   we should do that
> > > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > > demoting
> > > > > > > APIs
> > > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> > any
> > > > > > version
> > > > > > > > bump,
> > > > > > > > >   are not good practices.
> > > > > > > > >   3. Ideally, with this FLIP, developers should be more
> > careful
> > > > and
> > > > > > > plan
> > > > > > > > >   API changes ahead. That means:
> > > > > > > > >      1. Be more careful with designing APIs and promoting
> > them
> > > to
> > > > > > > Public,
> > > > > > > > >      so that they won't be changed / removed very soon, and
> > > also
> > > > > the
> > > > > > > > >      maintenance overhead for keeping them after
> deprecation
> > > > should
> > > > > > be
> > > > > > > > >      affordable. (I believe this also aligns with John's
> > > > opinion.)
> > > > > > > > >      2. Plan the deprecation / removal of old APIs that we
> > > don't
> > > > > want
> > > > > > > to
> > > > > > > > >      carry to the next major version earlier, so that
> they'll
> > > > have
> > > > > > the
> > > > > > > > >      2-minor-release migration period before the next major
> > > > version
> > > > > > > bump.
> > > > > > > > >   4. A practical situation is that, we are not ready for
> > > > > deprecating
> > > > > > > the
> > > > > > > > >   DataStream API, nor afford to carry it for another couple
> > of
> > > > > years.
> > > > > > > > >      1. We cannot deprecate it now, because the new
> > > > ProcessFunction
> > > > > > API
> > > > > > > > >      (the planned replacement for DataStream API) is not
> yet
> > > > ready.
> > > > > > We
> > > > > > > > haven't
> > > > > > > > >      planned the ProcessFunction API earlier because the
> > > original
> > > > > > > > > plan was to do
> > > > > > > > >      in-place changes directly on the DataStream API in
> > release
> > > > > 2.0,
> > > > > > > > until the
> > > > > > > > >      smooth migration period is proposed. Before this, the
> > only
> > > > > > > > guarantees we
> > > > > > > > >      provide for Public APIs is that they'll stay
> compatible
> > > > until
> > > > > > the
> > > > > > > > next
> > > > > > > > >      major version, which led to the understanding the
> > in-place
> > > > > > changes
> > > > > > > > on
> > > > > > > > >      Public APIs should be fine.
> > > > > > > > >      2. We also cannot afford carrying the DataStream API
> for
> > > > > another
> > > > > > > > >      couple of years. One of the reasons that we want to
> > > > refactor /
> > > > > > > > replace
> > > > > > > > >      DataStream API is that it exposes and depends on too
> > many
> > > > > > > > > Flink's internal
> > > > > > > > >      runtime implementations, which has already limited /
> > > blocked
> > > > > > plans
> > > > > > > > for
> > > > > > > > >      Flink's internal improvements for many times. Also due
> > to
> > > > the
> > > > > > wide
> > > > > > > > and
> > > > > > > > >      transitive dependencies on the internals, maintaining
> > two
> > > > sets
> > > > > > of
> > > > > > > > lower
> > > > > > > > >      APIs itself is expensive and would be better to keep
> it
> > > > short.
> > > > > > > This
> > > > > > > > is
> > > > > > > > >      admittedly a historical design flaw which should be
> > > avoided
> > > > in
> > > > > > > > future.
> > > > > > > > >
> > > > > > > > > What we haven't reached consensus is about how to deal with
> > the
> > > > > > current
> > > > > > > > > situation around the DataStream API. There are two options.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   1. We say this FLIP is enforced starting release 2.0. For
> > > > current
> > > > > > 1.x
> > > > > > > > >   APIs, we provide a migration period with best effort,
> while
> > > > > > allowing
> > > > > > > > >   exceptions for immediate removal in 2.0. That means we
> will
> > > > still
> > > > > > try
> > > > > > > > with
> > > > > > > > >   best effort to get the ProcessFuncion API ready and
> > deprecate
> > > > the
> > > > > > > > >   DataStream API in 1.x, but will also be allowed to remove
> > > > > > DataStream
> > > > > > > > API in
> > > > > > > > >   2.0 if it's not deprecated 2 minor releases before the
> > major
> > > > > > version
> > > > > > > > bump.
> > > > > > > > >   2. We strictly follow the process in this FLIP, and will
> > > > quickly
> > > > > > bump
> > > > > > > > >   the major version from 2.x to 3.0 once the migration
> period
> > > for
> > > > > > > > DataStream
> > > > > > > > >   API is reached.
> > > > > > > > >
> > > > > > > > > I'm personally in favor of option 1. As the
> migration-period
> > > rule
> > > > > is
> > > > > > > > newly
> > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > where
> > > we
> > > > > > > already
> > > > > > > > > missed the best chance for planning the deprecation.
> > Moreover,
> > > I
> > > > do
> > > > > > > > believe
> > > > > > > > > having a quick major version bump does confuse users. Yes,
> we
> > > can
> > > > > > > explain
> > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> anything
> > > > other
> > > > > > than
> > > > > > > > the
> > > > > > > > > removal of an deprecated API. But having to explain this
> > itself
> > > > is
> > > > > an
> > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > understanding,
> > > > > > his
> > > > > > > > > major point is that a quick major version bump causes
> barely
> > > any
> > > > > > actual
> > > > > > > > > lose on users, while in option 1 not providing the
> migration
> > > > period
> > > > > > or
> > > > > > > > > providing a shorter on is an actual lose on users.
> (@Becket,
> > > > please
> > > > > > > > correct
> > > > > > > > > me if I'm mistaken.)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > And we'd like to hear more feedback from the community.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Xintong
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > > > vvcephei@apache.org
> > > > > > > > <ma...@apache.org>> wrote:
> > > > > > > > >
> > > > > > > > >> Hi Becket and Xintong,
> > > > > > > > >>
> > > > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > > > >>
> > > > > > > > >> Once an API is marked Public, we're committing to support
> it
> > > > until
> > > > > > > it's
> > > > > > > > >> deprecated, and once it's deprecated, to leave it in place
> > for
> > > > at
> > > > > > > least
> > > > > > > > two
> > > > > > > > >> minor releases and then only remove it in the following
> > major
> > > > > > release.
> > > > > > > > >>
> > > > > > > > >> As a user, I would be dismayed to discover that the
> project
> > > > > > > maintainers
> > > > > > > > >> demoted a Public API to PublicEvolving or Experimental in
> > > order
> > > > to
> > > > > > > > violate
> > > > > > > > >> the spirit of the deprecation period for Public APIs.
> > > Likewise,
> > > > > I'd
> > > > > > > > view a
> > > > > > > > >> rapid sequence of minor releases for the purpose of
> dropping
> > > > > > > deprecated
> > > > > > > > >> APIs in the next major release as another violation of the
> > > > spirit
> > > > > of
> > > > > > > our
> > > > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > > > >>
> > > > > > > > >> I'm glad we're gaming out these situations before we
> > institute
> > > > > this
> > > > > > > > FLIP,
> > > > > > > > >> and I hope we all converge on a clear understanding of
> what
> > > > we're
> > > > > > > > >> guaranteeing. From where I'm sitting, the essence of this
> > FLIP
> > > > is
> > > > > > > > >> specifically to prevent us from doing whatever we want and
> > > > instead
> > > > > > to
> > > > > > > > >> commit to providing a suitable notice period to users who
> > rely
> > > > on
> > > > > > > Public
> > > > > > > > >> APIs.
> > > > > > > > >>
> > > > > > > > >> It honestly shouldn't matter what the maintenance overhead
> > is
> > > > (or,
> > > > > > > > rather,
> > > > > > > > >> evaluating the maintenance overhead should be a
> > consideration
> > > > > before
> > > > > > > we
> > > > > > > > >> promote an API to Public to begin with). My experience in
> > > other
> > > > > > > > projects is
> > > > > > > > >> that regretting a feature so much you want to drop it asap
> > is
> > > > > > > extremely
> > > > > > > > >> rare, and that living with the pain for a little while
> until
> > > the
> > > > > > > > >> deprecation period expires will probably make us better
> > > > engineers.
> > > > > > > > >>
> > > > > > > > >> With that all said, one thing that is not a violation is
> to
> > > > > propose
> > > > > > a
> > > > > > > > new
> > > > > > > > >> major release in order to drop especially burdensome
> > > deprecated
> > > > > APIs
> > > > > > > > that
> > > > > > > > >> have already enjoyed their full deprecation period. Those
> > > really
> > > > > bad
> > > > > > > > "case
> > > > > > > > >> A" features will be very rare.
> > > > > > > > >>
> > > > > > > > >> Looking over the FLIP again, I see a reference to the
> > release
> > > > > > > > >> documentation, but not to the build tooling support for
> the
> > > > > > Deprecated
> > > > > > > > >> annotation, so it might help to share this scenario: The
> way
> > > > this
> > > > > > will
> > > > > > > > all
> > > > > > > > >> come together in practice is that, by giving good
> stability
> > > > > > > guarantees,
> > > > > > > > we
> > > > > > > > >> will encourage our users to bump minor versions when they
> > are
> > > > > > > released,
> > > > > > > > >> which means that they will see compiler warnings
> immediately
> > > if
> > > > > any
> > > > > > of
> > > > > > > > the
> > > > > > > > >> APIs they've used are deprecated. In fact, I'd recommend
> for
> > > > them
> > > > > to
> > > > > > > use
> > > > > > > > >> the `-Werror` javac flag to be sure that they don't miss
> > those
> > > > > > > warnings.
> > > > > > > > >> Given our release cadence, this means that they will have
> > > about
> > > > > six
> > > > > > > > months
> > > > > > > > >> to migrate off of any deprecated APIs without being forced
> > to
> > > > > avoid
> > > > > > > > >> updates. And they really need to be able to adopt further
> > > minor
> > > > > > > updates
> > > > > > > > in
> > > > > > > > >> the series, since they may contain further improvements
> that
> > > > > > actually
> > > > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > > > >>
> > > > > > > > >> The compiler warnings are a far more reliable mechanism to
> > > > > advertise
> > > > > > > > >> deprecation than release notes, since users' own builds
> will
> > > > > notify
> > > > > > > them
> > > > > > > > >> right away about only the deprecations that are relevant
> to
> > > > them,
> > > > > > but
> > > > > > > it
> > > > > > > > >> does rely on users feeling ok to bump up minor versions
> in a
> > > > > timely
> > > > > > > > >> fashion. If we consistently burn them by not observing the
> > > > > stability
> > > > > > > > >> guarantees, they'll try to avoid updates instead, and the
> > > whole
> > > > > > thing
> > > > > > > > falls
> > > > > > > > >> apart.
> > > > > > > > >>
> > > > > > > > >> Thanks again,
> > > > > > > > >> -John
> > > > > > > > >>
> > > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > > > >>> Hi Xintong,
> > > > > > > > >>>
> > > > > > > > >>> Let's compare the following cases:
> > > > > > > > >>>
> > > > > > > > >>> A. If the maintenance overhead of the deprecated API is
> > high,
> > > > and
> > > > > > we
> > > > > > > > want
> > > > > > > > >>> to remove it after two minor releases. Then there are two
> > > > > options:
> > > > > > > > >>>    A1: Demote the API in the major version bump and
> remove
> > > the
> > > > > code
> > > > > > > > >> with a
> > > > > > > > >>> minor version bump.
> > > > > > > > >>>    A2: Do exactly the same as 1-A, except that when
> > removing
> > > > the
> > > > > > > code,
> > > > > > > > >>> bump up the major version. So this might be a short major
> > > > > version.
> > > > > > > > >>>
> > > > > > > > >>> B. If the maintenance overhead of the deprecated API is
> not
> > > > high,
> > > > > > > > >> therefore
> > > > > > > > >>> keeping it for a long time is affordable. There are also
> > two
> > > > > > options:
> > > > > > > > >>>    B1: Same as A-1, demote the API in the major version
> > bump
> > > > and
> > > > > > > remove
> > > > > > > > >>> the code with a minor version bump.
> > > > > > > > >>>    B2: Keep the API for all the minor versions in the
> major
> > > > > > version,
> > > > > > > > and
> > > > > > > > >>> only remove the code in the next major version.
> > > > > > > > >>>
> > > > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > > > >>>
> > > > > > > > >>> For case A:
> > > > > > > > >>> The following stays the same for A1 and A2:
> > > > > > > > >>>  - users will lose the API after two minor leases. So the
> > > > > migration
> > > > > > > > >> period
> > > > > > > > >>> is the same for A-1 and A-2.
> > > > > > > > >>>  - A1 and A2 will have exactly the same code after the
> > > removal
> > > > of
> > > > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > > > >>>  - To move forward, users need to move to the next minor
> > > > version
> > > > > in
> > > > > > > A1,
> > > > > > > > >> or
> > > > > > > > >>> to the next major version in A2. Because the code are the
> > > same,
> > > > > the
> > > > > > > > >> actual
> > > > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > > > >>>
> > > > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > > > >>>  - A1 allows keeping the major version release cadence.
> A-2
> > > > will
> > > > > > > have a
> > > > > > > > >>> short major version release.
> > > > > > > > >>>  - A1 breaks the well understood API semantic, while A-2
> > does
> > > > > not.
> > > > > > > > >>>
> > > > > > > > >>> From what I see, since there is no well establish
> standard
> > > > > > regarding
> > > > > > > > how
> > > > > > > > >>> long a major version should be released, A short major
> > > version
> > > > > > > release
> > > > > > > > is
> > > > > > > > >>> potential and emotional. It is not ideal but does not
> have
> > > > > material
> > > > > > > > >>> downsides compared with A1. I did not hear anyone
> > complaining
> > > > > about
> > > > > > > > Kafka
> > > > > > > > >>> only has two 1.x release. However, A1 actually breaks the
> > > well
> > > > > > > > understood
> > > > > > > > >>> API semantic, which has more material impact.
> > > > > > > > >>>
> > > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should
> > fall
> > > > into
> > > > > > > case
> > > > > > > > B.
> > > > > > > > >>> So, short major versions should be very occasional. I'd
> be
> > > very
> > > > > > > > concerned
> > > > > > > > >>> if the reason we choose A1 is simply because we cannot
> > afford
> > > > > > > > >> maintaining a
> > > > > > > > >>> bunch of deprecated APIs until the next major version.
> This
> > > > > > indicates
> > > > > > > > >> that
> > > > > > > > >>> the actual problem we need to solve is to lower the
> > > maintenance
> > > > > > > > overhead
> > > > > > > > >> of
> > > > > > > > >>> deprecated APIs, so that we are comfortable to keep them
> > > > longer.
> > > > > As
> > > > > > > > John
> > > > > > > > >>> and I mentioned earlier, there are ways to achieve this
> and
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > >>> learn how to do it in Flink. Otherwise, our discussion
> > about
> > > > > > > versioning
> > > > > > > > >>> here does not bring much value, because we will end up
> > with a
> > > > > bunch
> > > > > > > of
> > > > > > > > >>> short-lived APIs which upset our users, no matter how we
> > > > version
> > > > > > the
> > > > > > > > >>> releases.
> > > > > > > > >>>
> > > > > > > > >>> So, if there are concrete examples that you think will
> > block
> > > us
> > > > > > from
> > > > > > > > >>> keeping API stability with affordable cost, let's take a
> > look
> > > > > > > together
> > > > > > > > >> and
> > > > > > > > >>> see if that can be improved.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>>
> > > > > > > > >>> Jiangjie (Becket) Qin
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> From what I see,
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > > > tonysong820@gmail.com
> > > > > > > > <ma...@gmail.com>>
> > > > > > > > >> wrote:
> > > > > > > > >>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> The part I don't understand is if we are willing to
> have
> > a
> > > > > > > migration
> > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > what
> > > do
> > > > > we
> > > > > > > > >> lose to
> > > > > > > > >>>> do
> > > > > > > > >>>>> a major version bump instead, so we don't break the
> > common
> > > > > > > versioning
> > > > > > > > >>>>> semantic?
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>> I think we are talking about the cases where a major
> > version
> > > > > bump
> > > > > > > > >> happens
> > > > > > > > >>>> before a deprecated Public API reaches its migration
> > period.
> > > > So
> > > > > > > > >> removing
> > > > > > > > >>>> the API with another major version bump means we have
> two
> > > > > > > consecutive
> > > > > > > > >> major
> > > > > > > > >>>> versions in a very short time. And as previously
> > mentioned,
> > > > > having
> > > > > > > > >> frequent
> > > > > > > > >>>> major version bumps would weaken the value of the
> > commitment
> > > > > > "Public
> > > > > > > > >> API
> > > > > > > > >>>> stay compatible within a major version". I think users
> > also
> > > > have
> > > > > > > > >>>> expectations about how long a major version should live,
> > > which
> > > > > > > should
> > > > > > > > >> be at
> > > > > > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > > > > > >>>>
> > > > > > > > >>>> This is another option, but I think it is very likely
> more
> > > > > > expensive
> > > > > > > > >> than
> > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > will
> > > be
> > > > > > > > >> questions
> > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> in
> > > > 2.0"?
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Yes, it's more expensive than simply bumping the major
> > > > version,
> > > > > > but
> > > > > > > > >> IMHO
> > > > > > > > >>>> it's probably cheaper than carrying the API until the
> next
> > > > major
> > > > > > > > >> version if
> > > > > > > > >>>> we don't bump the major version very soon. And see my
> > reply
> > > > > above
> > > > > > on
> > > > > > > > >> why
> > > > > > > > >>>> not bumping immediately.
> > > > > > > > >>>>
> > > > > > > > >>>> Best,
> > > > > > > > >>>>
> > > > > > > > >>>> Xintong
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > > > becket.qin@gmail.com
> > > > > > > > <ma...@gmail.com>>
> > > > > > > > >> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hi Xintong,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Please see the replies below.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I see your point, that users would feel surprised if
> they
> > > > find
> > > > > > > > >> things no
> > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> release.
> > > > > > However,
> > > > > > > > >> I'd
> > > > > > > > >>>>> like
> > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > similar
> > > > > > > > >> problem
> > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > attention
> > > > > and
> > > > > > > > >> make
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> longer
> > > be
> > > > > > Public
> > > > > > > > >> in
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > > warnings
> > > > > > in
> > > > > > > > >>>> logs,
> > > > > > > > >>>>>> etc.
> > > > > > > > >>>>>
> > > > > > > > >>>>> First of all, I am not a fan of removing PublicEvolving
> > > APIs
> > > > in
> > > > > > > minor
> > > > > > > > >>>>> version changes. Personally speaking, I want them to be
> > > also
> > > > > > > removed
> > > > > > > > >> in
> > > > > > > > >>>>> major version changes.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Empirically, I rarely see projects with complex API
> > > semantic
> > > > > work
> > > > > > > > >> well.
> > > > > > > > >>>>> Many, if not most, users don't read docs / release
> notes
> > /
> > > > > > warnings
> > > > > > > > >> in
> > > > > > > > >>>>> logs. I don't think that is their fault. Typically an
> > > > > application
> > > > > > > > >>>> developer
> > > > > > > > >>>>> will have to deal with dozens of libraries maintained
> by
> > > > > various
> > > > > > > > >>>>> communities / groups. It is just too difficult for them
> > to
> > > > keep
> > > > > > > > >> track of
> > > > > > > > >>>>> all the specific semantics each project puts there. If
> > you
> > > > > think
> > > > > > > > >> about
> > > > > > > > >>>> it,
> > > > > > > > >>>>> how many of Flink developers read all the release notes
> > of
> > > > > Guava,
> > > > > > > > >> Apache
> > > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a
> > > > version
> > > > > > of
> > > > > > > > >> them?
> > > > > > > > >>>>> One would probably try bumping the dependency version
> and
> > > see
> > > > > if
> > > > > > > > >> there is
> > > > > > > > >>>>> an exception and solve them case by case. And if it is
> a
> > > > minor
> > > > > > > > >> version
> > > > > > > > >>>>> bump, one probably does not expect an exception at all.
> > > > Another
> > > > > > > > >> example
> > > > > > > > >>>> is
> > > > > > > > >>>>> that in Flink we still have so many usage of deprecated
> > > > methods
> > > > > > all
> > > > > > > > >> over
> > > > > > > > >>>>> the place, and I strongly doubt everyone knows when the
> > > > source
> > > > > > code
> > > > > > > > >> of
> > > > > > > > >>>>> these methods are deprecated and when they should be
> > > removed.
> > > > > > > > >>>>>
> > > > > > > > >>>>> So, the most important principle of API is simple and
> > > > > intuitive.
> > > > > > > The
> > > > > > > > >>>>> versioning semantic is a simple and universally
> accepted
> > > API
> > > > > > > > >> stability
> > > > > > > > >>>>> standard. If we ourselves as Flink developers are
> relying
> > > on
> > > > > this
> > > > > > > > >> for our
> > > > > > > > >>>>> own dependencies. I don't think we can expect more from
> > our
> > > > > > users.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Another possible alternative: whenever there's a
> > deprecated
> > > > > > Public
> > > > > > > > >> API
> > > > > > > > >>>> that
> > > > > > > > >>>>>> reaches a major version bump before the migration
> > period,
> > > > and
> > > > > we
> > > > > > > > >> also
> > > > > > > > >>>>> don't
> > > > > > > > >>>>>> want to carry it for all the next major release
> series,
> > we
> > > > may
> > > > > > > > >> consider
> > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > version
> > > > > > after
> > > > > > > > >> the
> > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > then
> > > we
> > > > > > bump
> > > > > > > > >> to
> > > > > > > > >>>> 2.0,
> > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > provide
> > > > > > users
> > > > > > > > >>>> another
> > > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> > the
> > > > > > > > >>>> 2-minor-release
> > > > > > > > >>>>>> migration period.
> > > > > > > > >>>>>
> > > > > > > > >>>>> This is another option, but I think it is very likely
> > more
> > > > > > > expensive
> > > > > > > > >> than
> > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > will
> > > be
> > > > > > > > >> questions
> > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> in
> > > > 2.0"?
> > > > > > > > >>>>>
> > > > > > > > >>>>> I think my major point is, we should not carry APIs
> > > > deprecated
> > > > > > in a
> > > > > > > > >>>>>> previous major version along all the next major
> version
> > > > > series.
> > > > > > > I'd
> > > > > > > > >>>> like
> > > > > > > > >>>>> to
> > > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > > period,
> > > > > as
> > > > > > > > >> long
> > > > > > > > >>>> as
> > > > > > > > >>>>> it
> > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> it
> > > > > doesn't
> > > > > > > > >> work,
> > > > > > > > >>>> I'd
> > > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > > fallback
> > > > > > to
> > > > > > > > >> only
> > > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > > >>>>>
> > > > > > > > >>>>> The part I don't understand is if we are willing to
> have
> > a
> > > > > > > migration
> > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > what
> > > do
> > > > > we
> > > > > > > > >> lose to
> > > > > > > > >>>> do
> > > > > > > > >>>>> a major version bump instead, so we don't break the
> > common
> > > > > > > versioning
> > > > > > > > >>>>> semantic?
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > > > tonysong820@gmail.com
> > > > > > > > <ma...@gmail.com>>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> As an end user who only uses Public APIs, if I don't
> > > change
> > > > > my
> > > > > > > > >> code
> > > > > > > > >>>> at
> > > > > > > > >>>>>>> all, my expectation is the following:
> > > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> issue, I
> > > am
> > > > > fine
> > > > > > > > >> with
> > > > > > > > >>>>> all
> > > > > > > > >>>>>>> the 2.x versions.
> > > > > > > > >>>>>>> Actually I think there are some dependency version
> > > > resolution
> > > > > > > > >>>> policies
> > > > > > > > >>>>>> out
> > > > > > > > >>>>>>> there which picks the highest minor version when the
> > > > > > dependencies
> > > > > > > > >>>> pull
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> multiple minor versions of the same jar, which may be
> > > > broken
> > > > > if
> > > > > > > > >> we
> > > > > > > > >>>>> remove
> > > > > > > > >>>>>>> the API in minor releases.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I see your point, that users would feel surprised if
> > they
> > > > find
> > > > > > > > >> things
> > > > > > > > >>>> no
> > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> release.
> > > > > > However,
> > > > > > > > >> I'd
> > > > > > > > >>>>> like
> > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > similar
> > > > > > > > >> problem
> > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > attention
> > > > > and
> > > > > > > > >> make
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> longer
> > > be
> > > > > > Public
> > > > > > > > >> in
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > > warnings
> > > > > > in
> > > > > > > > >>>> logs,
> > > > > > > > >>>>>> etc.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Another possible alternative: whenever there's a
> > > deprecated
> > > > > > Public
> > > > > > > > >> API
> > > > > > > > >>>>> that
> > > > > > > > >>>>>> reaches a major version bump before the migration
> > period,
> > > > and
> > > > > we
> > > > > > > > >> also
> > > > > > > > >>>>> don't
> > > > > > > > >>>>>> want to carry it for all the next major release
> series,
> > we
> > > > may
> > > > > > > > >> consider
> > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > version
> > > > > > after
> > > > > > > > >> the
> > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > then
> > > we
> > > > > > bump
> > > > > > > > >> to
> > > > > > > > >>>> 2.0,
> > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > provide
> > > > > > users
> > > > > > > > >>>> another
> > > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> > the
> > > > > > > > >>>> 2-minor-release
> > > > > > > > >>>>>> migration period.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I think my major point is, we should not carry APIs
> > > > deprecated
> > > > > > in
> > > > > > > a
> > > > > > > > >>>>>> previous major version along all the next major
> version
> > > > > series.
> > > > > > > I'd
> > > > > > > > >>>> like
> > > > > > > > >>>>> to
> > > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > > period,
> > > > > as
> > > > > > > > >> long
> > > > > > > > >>>> as
> > > > > > > > >>>>> it
> > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> it
> > > > > doesn't
> > > > > > > > >> work,
> > > > > > > > >>>> I'd
> > > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > > fallback
> > > > > > to
> > > > > > > > >> only
> > > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Best,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Xintong
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > > > vvcephei@apache.org
> > > > > > > > <ma...@apache.org>
> > > > > > > > >>>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hi Becket,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> > > conversation
> > > > > > about
> > > > > > > > >>>>>>> compatibility outside of this FLIP thread, but for
> > now, I
> > > > can
> > > > > > > > >> accept
> > > > > > > > >>>>> your
> > > > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks again,
> > > > > > > > >>>>>>> John
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > > >>>>>>>> Hi John,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in
> major
> > > > > > releases
> > > > > > > > >>>>> across
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > PublicEvolving
> > > > > APIs
> > > > > > > > >> are
> > > > > > > > >>>> by
> > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> should
> > > be
> > > > > > > > >> reflected
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> must
> > > keep
> > > > > > them
> > > > > > > > >>>>> around
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > > drop
> > > > > them
> > > > > > > > >> in a
> > > > > > > > >>>>>>> minor or
> > > > > > > > >>>>>>>>> patch release.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Personally speaking, I would love to do this, for
> > > exactly
> > > > > the
> > > > > > > > >>>> reason
> > > > > > > > >>>>>> you
> > > > > > > > >>>>>>>> mentioned. However, I did not propose this due to
> the
> > > > > > following
> > > > > > > > >>>>>> reasons:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> > > > accepted
> > > > > > > > >> FLIPs
> > > > > > > > >>>> too
> > > > > > > > >>>>>>> soon.
> > > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > > > development.
> > > > > At
> > > > > > > > >> this
> > > > > > > > >>>>>>> point,
> > > > > > > > >>>>>>>> Flink still lacks some design / routines to support
> > good
> > > > API
> > > > > > > > >>>>>>> evolvability /
> > > > > > > > >>>>>>>> extensibility. Just like you said, it takes some
> time
> > to
> > > > be
> > > > > > > > >> good at
> > > > > > > > >>>>>> this.
> > > > > > > > >>>>>>>> In this case, my concern is that only removing
> > > > Experimental
> > > > > /
> > > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may
> > result
> > > in
> > > > > too
> > > > > > > > >> much
> > > > > > > > >>>>>>>> overhead and dramatically slow down the development
> of
> > > > > Flink.
> > > > > > > > >> So, I
> > > > > > > > >>>>> was
> > > > > > > > >>>>>>>> thinking that we can start with the current status.
> > > > > Hopefully
> > > > > > > > >> after
> > > > > > > > >>>>> we
> > > > > > > > >>>>>>> are
> > > > > > > > >>>>>>>> more comfortable with the maintenance overhead of
> > > > deprecated
> > > > > > > > >> APIs,
> > > > > > > > >>>> we
> > > > > > > > >>>>>> can
> > > > > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > > > > >> PublicEvolving
> > > > > > > > >>>>> APIs.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Hi Becket,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process
> is
> > > > > really
> > > > > > > > >>>>>> important.
> > > > > > > > >>>>>>> I
> > > > > > > > >>>>>>>>> understand some people’s concerns about the
> > additional
> > > > > burden
> > > > > > > > >> for
> > > > > > > > >>>>>>> project
> > > > > > > > >>>>>>>>> maintainers, but my personal experience with Kafka
> > has
> > > > been
> > > > > > > > >> that
> > > > > > > > >>>>> it’s
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to
> > users.
> > > > In
> > > > > > > > >> fact,
> > > > > > > > >>>>> users
> > > > > > > > >>>>>>>>> being able to confidently upgrade is also a benefit
> > to
> > > > > > > > >>>> maintainers,
> > > > > > > > >>>>> as
> > > > > > > > >>>>>>> we
> > > > > > > > >>>>>>>>> will get fewer questions from people stuck on very
> > old
> > > > > > > > >> versions.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> One question:
> > > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in
> > major
> > > > > > > > >> releases
> > > > > > > > >>>>> across
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > PublicEvolving
> > > > > APIs
> > > > > > > > >> are
> > > > > > > > >>>> by
> > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> should
> > > be
> > > > > > > > >> reflected
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> must
> > > keep
> > > > > > them
> > > > > > > > >>>>> around
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > > drop
> > > > > them
> > > > > > > > >> in a
> > > > > > > > >>>>>>> minor or
> > > > > > > > >>>>>>>>> patch release.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The advantage of forbidding the removal of any API
> in
> > > > minor
> > > > > > or
> > > > > > > > >>>> patch
> > > > > > > > >>>>>>>>> releases is that users will get a strong guarantee
> > that
> > > > > they
> > > > > > > > >> can
> > > > > > > > >>>>> bump
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> minor or patch version and still be able to
> compile,
> > or
> > > > > even
> > > > > > > > >> just
> > > > > > > > >>>>>>> re-link
> > > > > > > > >>>>>>>>> and know that they won’t face “MethodDef”
> exceptions
> > at
> > > > run
> > > > > > > > >> time.
> > > > > > > > >>>>> This
> > > > > > > > >>>>>>> is a
> > > > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> > > Experimental
> > > > > > APIs
> > > > > > > > >>>>> outside
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>>> major releases, users can no longer confidently
> > > upgrade.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple
> of
> > > > > points:
> > > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation
> instead
> > of
> > > > > > > > >>>> introducing
> > > > > > > > >>>>>> our
> > > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > > > > > integrated
> > > > > > > > >>>> into
> > > > > > > > >>>>>>> build
> > > > > > > > >>>>>>>>> tools and IDEs.
> > > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this
> > > FLIP;
> > > > > it
> > > > > > > > >> seems
> > > > > > > > >>>>>>>>> orthogonal, and something that should probably be
> > taken
> > > > > > > > >>>> case-by-case
> > > > > > > > >>>>>>>>> anyway.
> > > > > > > > >>>>>>>>> * Aside from deprecation and removal, there have
> been
> > > > some
> > > > > > > > >>>>> discussions
> > > > > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible
> > > ways.
> > > > > > > > >> This is
> > > > > > > > >>>>>>> somewhat
> > > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it
> > > before,
> > > > > > it’ll
> > > > > > > > >>>> take
> > > > > > > > >>>>>> some
> > > > > > > > >>>>>>>>> time to become good at it. I feel like this topic
> > > should
> > > > > also
> > > > > > > > >> be
> > > > > > > > >>>>>>> orthogonal
> > > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to
> > > adopt a
> > > > > > > > >> simple
> > > > > > > > >>>>>> policy
> > > > > > > > >>>>>>> not
> > > > > > > > >>>>>>>>> to break existing user programs, and leave the
> “how”
> > up
> > > > to
> > > > > > > > >>>>>> implementers
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> reviewers.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > >>>>>>>>> John
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > > >>>>>>>>>> Hi All,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong
> > is
> > > a
> > > > > > > > >> great
> > > > > > > > >>>>> idea.
> > > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > > > > > >>>> @PublicEvolving
> > > > > > > > >>>>>> --(2
> > > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1
> major
> > > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot
> of
> > > > > > > > >> flexibility
> > > > > > > > >>>>>>> without
> > > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are
> allowed
> > to
> > > > > > > > >> change
> > > > > > > > >>>>>> between
> > > > > > > > >>>>>>>>> major
> > > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> > > > provides
> > > > > > > > >>>>> additional
> > > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> > > > themself
> > > > > > > > >> are
> > > > > > > > >>>> not
> > > > > > > > >>>>>>>>> changed.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> > > issues. I
> > > > > > > > >> want to
> > > > > > > > >>>>>> move
> > > > > > > > >>>>>>> one
> > > > > > > > >>>>>>>>>> step further and suggest introducing one more
> > > annotation
> > > > > > > > >>>> @Retired.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise
> of
> > > > > > > > >> downgrading
> > > > > > > > >>>>>>> @Public
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my
> reply,
> > > > Java
> > > > > > > > >>>> standard
> > > > > > > > >>>>>>>>>> @deprecated should be used in the early stage of
> the
> > > > > > > > >> deprecation
> > > > > > > > >>>>>>> process
> > > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since
> Java
> > > does
> > > > > not
> > > > > > > > >>>> allow
> > > > > > > > >>>>>> us
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>> extend annotation, I think it would be feasible to
> > > have
> > > > > the
> > > > > > > > >> new
> > > > > > > > >>>>>>> @Retired
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>> help us monitor and manage the deprecation
> process,
> > > > house
> > > > > > > > >>>>> cleaning,
> > > > > > > > >>>>>>> etc.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> @Retired:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > > > > > >> compatibility
> > > > > > > > >>>>>> before
> > > > > > > > >>>>>>>>> APIs
> > > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> > > > processes
> > > > > > > > >> on
> > > > > > > > >>>> the
> > > > > > > > >>>>> ML
> > > > > > > > >>>>>>> are
> > > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1
> > > minor
> > > > > > > > >> release
> > > > > > > > >>>>>>> sharply
> > > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check
> > > whether
> > > > > it
> > > > > > > > >> is
> > > > > > > > >>>>>>> possible).
> > > > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible
> to
> > > > remove
> > > > > > > > >> them.
> > > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> > > > > @Retired
> > > > > > > > >> are
> > > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and
> @Internal
> > > ->
> > > > > > > > >> @Retired
> > > > > > > > >>>>>> could
> > > > > > > > >>>>>>>>> also
> > > > > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > > > > >>>> dev-friendliness,
> > > > > > > > >>>>>> but
> > > > > > > > >>>>>>> not
> > > > > > > > >>>>>>>>>> mandatory.
> > > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to
> > > support
> > > > > the
> > > > > > > > >>>>>>> deprecation
> > > > > > > > >>>>>>>>>> process management. Further extension is possible,
> > > since
> > > > > the
> > > > > > > > >>>>>>> annotation
> > > > > > > > >>>>>>>>> is
> > > > > > > > >>>>>>>>>> built by us.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>>> Jing
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the
> replies
> > > > inline
> > > > > > > > >>>> below.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a
> > Public
> > > > API
> > > > > > > > >> is
> > > > > > > > >>>>> also a
> > > > > > > > >>>>>>>>> kind of
> > > > > > > > >>>>>>>>>>>> such change, just like removing one, which can
> > only
> > > > > > > > >> happen
> > > > > > > > >>>> with
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> demoting
> > > > > Public
> > > > > > > > >>>> APIs
> > > > > > > > >>>>>>>>> anytime,
> > > > > > > > >>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> before
> > > > > > > > >> reaching
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > demoting
> > > > > > > > >> would
> > > > > > > > >>>> be a
> > > > > > > > >>>>>>> weaker
> > > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> > upon
> > > > > > > > >> major
> > > > > > > > >>>>>> version
> > > > > > > > >>>>>>>>> bumps,
> > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > 2-minor-release
> > > > > > > > >>>>>> migration
> > > > > > > > >>>>>>>>>>> period.
> > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > `@PublicEvolving`
> > > > > > > > >> sounds
> > > > > > > > >>>>>>> against
> > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> if
> > > > > > > > >> necessary,
> > > > > > > > >>>>>> e.g.,
> > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I
> > don't
> > > > > > > > >> change my
> > > > > > > > >>>>> code
> > > > > > > > >>>>>>> at
> > > > > > > > >>>>>>>>> all,
> > > > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > > issue, I
> > > > > am
> > > > > > > > >> fine
> > > > > > > > >>>>>> with
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > > > >>>>>>>>>>> Actually I think there are some dependency
> version
> > > > > > > > >> resolution
> > > > > > > > >>>>>>> policies
> > > > > > > > >>>>>>>>> out
> > > > > > > > >>>>>>>>>>> there which picks the highest minor version when
> > the
> > > > > > > > >>>> dependencies
> > > > > > > > >>>>>>> pull
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which
> may
> > be
> > > > > > > > >> broken if
> > > > > > > > >>>>> we
> > > > > > > > >>>>>>>>> remove
> > > > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > "legal"
> > > > > that
> > > > > > > > >> we
> > > > > > > > >>>>> bump
> > > > > > > > >>>>>> up
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > needed.
> > > > > > > > >> However,
> > > > > > > > >>>>> this
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> > APIs
> > > > > will
> > > > > > > > >>>> stay
> > > > > > > > >>>>>>> stable
> > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> can
> > > end
> > > > > > > > >>>> anytime.
> > > > > > > > >>>>>>> IMHO,
> > > > > > > > >>>>>>>>>>> short
> > > > > > > > >>>>>>>>>>>> major release series are not something "make the
> > end
> > > > > > > > >> users
> > > > > > > > >>>>>> happy",
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> make
> > > > > > > > >> frequent
> > > > > > > > >>>>>> breaking
> > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > approach,
> > > > we
> > > > > > > > >> can
> > > > > > > > >>>>>> still
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > > Public
> > > > > > > > >> APIs
> > > > > > > > >>>>>>>>> deprecated
> > > > > > > > >>>>>>>>>>> at
> > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > removed
> > > in
> > > > > > > > >> the
> > > > > > > > >>>> next
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I totally agree that frequent major version bumps
> > are
> > > > not
> > > > > > > > >>>> ideal,
> > > > > > > > >>>>>> but
> > > > > > > > >>>>>>>>> here
> > > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump
> which
> > > > > > > > >> removes a
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API.
> > > > > > > > >>>>>>>>>>> So the context is that we have already decided to
> > > > remove
> > > > > > > > >> this
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>> API
> > > > > > > > >>>>>>>>>>> while keeping everything else backwards
> > compatible. I
> > > > > > > > >> think a
> > > > > > > > >>>>> major
> > > > > > > > >>>>>>>>> version
> > > > > > > > >>>>>>>>>>> bump is a commonly understood signal here,
> compared
> > > > with
> > > > > a
> > > > > > > > >>>> minor
> > > > > > > > >>>>>>> version
> > > > > > > > >>>>>>>>>>> change. From end users' perspective, for those
> who
> > > are
> > > > > not
> > > > > > > > >>>>>> impacted,
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>> this case upgrading a major version is not
> > > necessarily
> > > > > more
> > > > > > > > >>>>>> involved
> > > > > > > > >>>>>>>>> than
> > > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as
> > smooth
> > > > as a
> > > > > > > > >>>>>> dependency
> > > > > > > > >>>>>>>>>>> version change. For those who are impacted, they
> > will
> > > > > lose
> > > > > > > > >> the
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API
> > > > > > > > >>>>>>>>>>> anyways and a major version bump ensures there is
> > no
> > > > > > > > >> surprise.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Public API is a well defined common concept, and
> > one
> > > > of
> > > > > > > > >> its
> > > > > > > > >>>>>>>>>>>>> convention is that it only changes with a major
> > > > version
> > > > > > > > >>>>> change.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a
> > > Public
> > > > > > > > >> API is
> > > > > > > > >>>>>> also a
> > > > > > > > >>>>>>>>> kind
> > > > > > > > >>>>>>>>>>>> of such change, just like removing one, which
> can
> > > only
> > > > > > > > >> happen
> > > > > > > > >>>>>> with
> > > > > > > > >>>>>>>>> major
> > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> demoting
> > > > > Public
> > > > > > > > >>>> APIs
> > > > > > > > >>>>>>>>> anytime,
> > > > > > > > >>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> before
> > > > > > > > >> reaching
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > demoting
> > > > > > > > >> would
> > > > > > > > >>>> be a
> > > > > > > > >>>>>>> weaker
> > > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> > upon
> > > > > > > > >> major
> > > > > > > > >>>>>> version
> > > > > > > > >>>>>>>>> bumps,
> > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > 2-minor-release
> > > > > > > > >>>>>> migration
> > > > > > > > >>>>>>>>>>> period.
> > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > `@PublicEvolving`
> > > > > > > > >> sounds
> > > > > > > > >>>>>>> against
> > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> if
> > > > > > > > >> necessary,
> > > > > > > > >>>>>> e.g.,
> > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> But it should be
> > > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if
> we
> > > > really
> > > > > > > > >> want
> > > > > > > > >>>>> to
> > > > > > > > >>>>>>> get
> > > > > > > > >>>>>>>>> rid
> > > > > > > > >>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > > "legal"
> > > > > > > > >> that we
> > > > > > > > >>>>>> bump
> > > > > > > > >>>>>>> up
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > needed.
> > > > > > > > >> However,
> > > > > > > > >>>>> this
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> > APIs
> > > > > will
> > > > > > > > >>>> stay
> > > > > > > > >>>>>>> stable
> > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> can
> > > end
> > > > > > > > >>>> anytime.
> > > > > > > > >>>>>>> IMHO,
> > > > > > > > >>>>>>>>>>> short
> > > > > > > > >>>>>>>>>>>> major release series are not something "make the
> > end
> > > > > > > > >> users
> > > > > > > > >>>>>> happy",
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> make
> > > > > > > > >> frequent
> > > > > > > > >>>>>> breaking
> > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > approach,
> > > > we
> > > > > > > > >> can
> > > > > > > > >>>>>> still
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > > Public
> > > > > > > > >> APIs
> > > > > > > > >>>>>>>>> deprecated
> > > > > > > > >>>>>>>>>>> at
> > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > removed
> > > in
> > > > > > > > >> the
> > > > > > > > >>>> next
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Given our track record I would prefer a regular
> > > cycle
> > > > > > > > >> (1-2
> > > > > > > > >>>>> years)
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> not
> > > put
> > > > > it
> > > > > > > > >>>> again
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > expectation
> > > > > on
> > > > > > > > >>>> when
> > > > > > > > >>>>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good
> > > time
> > > > > > > > >> for new
> > > > > > > > >>>>>> major
> > > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking
> changes
> > > > > > > > >> needed.
> > > > > > > > >>>> That
> > > > > > > > >>>>>>> makes
> > > > > > > > >>>>>>>>> 1-2
> > > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while
> > > > leaving
> > > > > > > > >> us
> > > > > > > > >>>> more
> > > > > > > > >>>>>>> time
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Xintong
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay
> Schepler
> > <
> > > > > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing
> > > minor
> > > > > > > > >>>>> versions
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure
> > how
> > > > > > > > >> viable
> > > > > > > > >>>> that
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>>>> practice.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially
> dropping
> > > > > > > > >> support
> > > > > > > > >>>> for
> > > > > > > > >>>>>> Java
> > > > > > > > >>>>>>>>> 8/11,
> > > > > > > > >>>>>>>>>>>>> which may very well be a problem for our
> current
> > > > users.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> > > confusion
> > > > > > > > >> about
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API
> > > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a
> > > mistake
> > > > > > > > >> in
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> NOTES
> > > > > > > > >>>>>>>>> part
> > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just
> > > fixed
> > > > > > > > >> it.
> > > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear
> that
> > > we
> > > > > > > > >> are on
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>> same
> > > > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if
> we
> > > > > > > > >> really
> > > > > > > > >>>> want
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> get
> > > > > > > > >>>>>>>>>>> rid
> > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took
> to
> > > get
> > > > us
> > > > > > > > >> to
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> ;)
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the
> > > > cadence
> > > > > > > > >> of
> > > > > > > > >>>>> major
> > > > > > > > >>>>>>>>> releases
> > > > > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> > > > opinions
> > > > > > > > >> on
> > > > > > > > >>>>> that.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor
> releases
> > > > > between
> > > > > > > > >>>> major
> > > > > > > > >>>>>>> ones,
> > > > > > > > >>>>>>>>>>>>> that for us means that users have to
> potentially
> > > deal
> > > > > > > > >> with
> > > > > > > > >>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular
> > > cycle
> > > > > > > > >> (1-2
> > > > > > > > >>>>>> years)
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> not
> > > put
> > > > > it
> > > > > > > > >>>> again
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > expectation
> > > > > on
> > > > > > > > >>>> when
> > > > > > > > >>>>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate
> > > thread.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in
> > > mind?
> > > > > > > > >> Not
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> I
> > > > > > > > >>>>>>>>> fully
> > > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change
> > here.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> > > performance
> > > > in
> > > > > > > > >>>>> certain
> > > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> > > unknowingly)
> > > > > > > > >> relied
> > > > > > > > >>>>> upon
> > > > > > > > >>>>>>>>>>>>> (
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > > > ),
> > > > > > > > a semantic change to some
> > > > > > > > >> API.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> > > submission.
> > > > A
> > > > > > > > >> few
> > > > > > > > >>>>>>> releases
> > > > > > > > >>>>>>>>> back
> > > > > > > > >>>>>>>>>>>>> we made changes such that the initialization of
> > the
> > > > job
> > > > > > > > >>>> master
> > > > > > > > >>>>>>>>> happens
> > > > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > > > >>>>>>>>>>>>> This meant the job submission call returns
> > sooner,
> > > > and
> > > > > > > > >> the
> > > > > > > > >>>> job
> > > > > > > > >>>>>>> state
> > > > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change,
> > but
> > > > the
> > > > > > > > >>>>> observed
> > > > > > > > >>>>>>>>> behavior
> > > > > > > > >>>>>>>>>>>>> may be different.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over
> time
> > we
> > > > > > > > >> changed
> > > > > > > > >>>>> what
> > > > > > > > >>>>>>> some
> > > > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
>
> Considering DataStream API is the most fundamental and complex API of
> Flink, I think it is worth a longer time than the general process for the
> deprecation period to wait for the new API be mature.
>

This inspires me. In this specific case, compared to how long should
DataStream API be removed after deprecation, it's probably more important
to answer the question how long would ProcessFunction API become mature and
stable after being introduced. According to FLIP-197[1], it requires 4
minor releases by default to promote an @Experimental API to @Public. And
for ProcessFunction API, which aims to replace DataStream API as one of the
most fundamental API of Flink, I'd expect this to take at least the default
time, or even longer. And we probably should wait until we believe
ProcessFunction API is stable to mark DataStream API as deprecated, rather
than as soon as it's introduced. Assuming we introduce the ProcessFunction
API in 2.0, that means we would need to wait for 6 minor releases (4 for
the new API to become stable, and 2 for the migration period) to remove
DataStream API, which is ~2.5 year (assuming 5 months / minor release),
which sounds acceptable for another major version bump.

To wrap things up, it seems to me, sadly, that anyway we cannot avoid the
overhead for maintaining both DataStream & ProcessFunction APIs for at
least 6 minor releases.

Best,

Xintong


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process



On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi all,
>
> Just want to make sure we are on the same page. There is another example[1]
> I was aware of recently that shows why more factors need to be taken care
> of than just the migration period. Thanks Galen for your hint.
>
> To put it simply, the concern about API deprecation is not that deprecated
> APIs have been removed too early (min migration period is required). The
> major concern is that APIs are marked as deprecated for a (too) long time,
> much longer than the migration period discussed in this thread, afaik.
> Since there is no clear picture/definition, no one knows when to do the
> migration for users(after the migration period has expired) and when to
> remove deprecated APIs for Flink developers.
>
> Based on all the information I knew, there are two kinds of obstacles that
> will and should block the deprecation process:
>
> 1. Lack of functionalities in new APIs. It happens e.g. with the
> SourceFunction to FLIP-27 Source migration. Users who rely on those
> functions can not migrate to new APIs.
> 2. new APIs have critical bugs. An example could be found at [1]. Users
> have to stick to the deprecated APIs.
>
> Since FLIP-321 is focusing on the API deprecation process, those blocking
> issues deserve attention and should be put into the FLIP. The current FLIP
> seems to only focus on migration periods. If we consider those blocking
> issues as orthogonal issues that are beyond the scope of this discussion,
> does it make sense to change the FLIP title to something like "Introduce
> minimum migration periods of API deprecation process"?
>
> Best regards,
> Jing
>
> [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
>
> On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <im...@gmail.com> wrote:
>
> > I agree with Jingsong and Becket.
> >
> > Look at the legacy SourceFunction (a small part of DataStream API),
> > the SourceFunction is still not and can't be marked deprecated[1] until
> > now after the new Source was released 2 years ago, because the new Source
> > still can't fully consume the abilities of legacy API. Considering
> > DataStream
> > API is the most fundamental and complex API of Flink, I think it is worth
> > a longer time than the general process for the deprecation period to
> > wait for the new API be mature. The above 2 options sound a bit of rush
> > for such a widely used API.
> >
> > I fully understand the concern of maintenance overhead, but it's a bit
> hard
> > for others to estimate maintenance costs without a concrete design and
> code
> > of the new ProcessFunction API. I agree with Becket that maybe we can
> > re-evaluate the API deprecation process once we have the new
> > ProcessFunction
> > API. If the maintenance is indeed huge, I think it is reasonable to have
> a
> > special rule for this case at that time.
> >
> > Best,
> > Jark
> >
> >
> > [1]: https://issues.apache.org/jira/browse/FLINK-28045
> >
> > On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com> wrote:
> >
> > > Hi Jingsong,
> > >
> > > Thanks for the reply. I completely agree with you.
> > >
> > > The above 2 options are based on the assumption that the community
> cannot
> > > afford to maintain the deprecated DataStream API for long. I'd say we
> > > should try everything we can to maintain it for as much time as
> possible.
> > > DataStream API is actually the most used API in Flink by so many users
> at
> > > this point. Removing it any time soon will dramatically hurt our users.
> > So
> > > ideally we should keep it for at least 2 years after deprecation, if
> not
> > > more.
> > >
> > > The prohibitively high maintenance overhead is just an assumption.
> > > Personally speaking, I don't feel this assumption is necessarily true.
> We
> > > should re-evaluate once we have the new ProcessFunction API in place.
> > > Without the code it is hard to tell for sure. I am actually kind of
> > > optimistic about the maintenance cost.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <ji...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Becket and all for your discussion.
> > > >
> > > > > 1. We say this FLIP is enforced starting release 2.0. For current
> 1.x
> > > > APIs,
> > > > we provide a migration period with best effort, while allowing
> > exceptions
> > > > for immediate removal in 2.0. That means we will still try with best
> > > effort
> > > > to get the ProcessFuncion API ready and deprecate the DataStream API
> in
> > > > 1.x, but will also be allowed to remove DataStream API in 2.0 if it's
> > not
> > > > deprecated 2 minor releases before the major version bump.
> > > >
> > > > > 2. We strictly follow the process in this FLIP, and will quickly
> bump
> > > the
> > > > major version from 2.x to 3.0 once the migration period for
> DataStream
> > > API
> > > > is reached.
> > > >
> > > > Sorry, I didn't read the previous detailed discussion because the
> > > > discussion list was so long.
> > > >
> > > > I don't really like either of these options.
> > > >
> > > > Considering that DataStream is such an important API, can we offer a
> > > third
> > > > option:
> > > >
> > > > 3. Maintain the DataStream API throughout 2.X and remove it until
> 3.x.
> > > But
> > > > there's no need to assume that 2.X is a short version, it's still a
> > > normal
> > > > major version.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> > > >
> > > > > Thanks much for the input, John, Stefan and Jing.
> > > > >
> > > > > I think Xingtong has well summarized the pros and cons of the two
> > > > options.
> > > > > Let's collect a few more opinions here and we can move forward with
> > the
> > > > one
> > > > > more people prefer.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks Xingtong for the summary. If I could only choose one of
> the
> > > > given
> > > > > > two options, I would go with option 1. I understood that option 2
> > > > worked
> > > > > > great with Kafka. But the bridge release will still confuse users
> > and
> > > > my
> > > > > > gut feeling is that many users will skip 2.0 and be waiting for
> 3.0
> > > or
> > > > > even
> > > > > > 3.x. And since fewer users will use Flink 2.x, the development
> > focus
> > > > will
> > > > > > be on Flink 3.0 with the fact that the current Flink release is
> > 1.17
> > > > and
> > > > > we
> > > > > > are preparing 2.0 release. That is weird for me.
> > > > > >
> > > > > > THB, I would not name the change from @Public to @Retired as a
> > > > demotion.
> > > > > > The purpose of @Retire is to extend the API lifecycle with one
> more
> > > > > stage,
> > > > > > like in the real world, people born, studied, graduated, worked,
> > and
> > > > > > retired. Afaiu from the previous discussion, there are two rules
> > we'd
> > > > > like
> > > > > > to follow simultaneously:
> > > > > >
> > > > > > 1. Public APIs can only be changed between major releases.
> > > > > > 2. A smooth migration phase should be offered to users, i.e. at
> > > least 2
> > > > > > minor releases after APIs are marked as @deprecated. There should
> > be
> > > > new
> > > > > > APIs as the replacement.
> > > > > >
> > > > > > Agree, those rules are good to improve the user friendliness.
> > Issues
> > > we
> > > > > > discussed are rising because we want to fulfill both of them. If
> we
> > > > take
> > > > > > care of deprecation very seriously, APIs can be marked as
> > > @Deprecated,
> > > > > only
> > > > > > when the new APIs as the replacement provide all functionalities
> > the
> > > > > > deprecated APIs have. In an ideal case without critical bugs that
> > > might
> > > > > > stop users adopting the new APIs. Otherwise the expected
> > > "replacement"
> > > > > will
> > > > > > not happen. Users will still stick to the deprecated APIs,
> because
> > > the
> > > > > new
> > > > > > APIs can not be used. For big features, it will need at least 4
> > minor
> > > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > > > >
> > > > > > - 1st minor release to build the new APIs as the replacement and
> > > > waiting
> > > > > > for feedback. It might be difficult to mark the old API as
> > deprecated
> > > > in
> > > > > > this release, because we are not sure if the new APIs could cover
> > > 100%
> > > > > > functionalities.
> > > > > > -  In the lucky case,  mark all old APIs as deprecated in the 2nd
> > > minor
> > > > > > release. (I would even suggest having the new APIs released at
> > least
> > > > for
> > > > > > two minor releases before marking it as deprecated to make sure
> > they
> > > > can
> > > > > > really replace the old APIs, in case we care more about smooth
> > > > migration)
> > > > > > - 3rd minor release for the migration period
> > > > > > -  In another lucky case, the 4th release is a major release, the
> > > > > > deprecated APIs could be removed.
> > > > > >
> > > > > > The above described scenario works only in an ideal case. In
> > reality,
> > > > it
> > > > > > might take longer to get the new APIs ready and mark the old API
> > > > > > deprecated. Furthermore, if the 4th release is not a major
> release,
> > > we
> > > > > will
> > > > > > have to maintain both APIs for many further minor releases. The
> > > > question
> > > > > is
> > > > > > how to know the next major release in advance, especially 4 minor
> > > > > releases'
> > > > > > period, i.e. more than 2 years in advance? Given that Flink
> > contains
> > > > many
> > > > > > modules, it is difficult to ask devs to create a 2-3 years
> > > deprecation
> > > > > plan
> > > > > > for each case. In case we want to build major releases at a fast
> > > pace,
> > > > > > let's say every two years, it means devs must plan any API
> > > deprecation
> > > > > > right after each major release. Afaiac, it is quite difficult.
> > > > > >
> > > > > > The major issue is, afaiu, if we follow rule 2, we have to keep
> all
> > > > > @Public
> > > > > > APIs, e.g. DataStream, that are not marked as deprecated yet, to
> > 2.0.
> > > > > Then
> > > > > > we have to follow rule 1 to keep it unchanged until we have 3.0.
> > That
> > > > is
> > > > > > why @Retired is useful to give devs more flexibility and still
> > > fulfill
> > > > > both
> > > > > > rules. Let's check it with examples:
> > > > > >
> > > > > > - we have @Public DataStream API in 1.18. It will not be marked
> > > > > > as @Deprecated, because the new APIs as the replacement are not
> > > ready.
> > > > > > - we keep the DataStream API itself unchanged in 2.0, but change
> > the
> > > > > > annotation from @Public to @Retire. New APIs will be introduced
> > too.
> > > In
> > > > > > this case, Rule 1 is ok, since public API is allowed to change
> > > between
> > > > > > major releases. Only changing annotation is the minimal change we
> > > could
> > > > > do
> > > > > > and it does not break rule 1. Rule 2 is ok too, since the
> > DataStream
> > > > APIs
> > > > > > work exactly the same.  Attention: the change of @Public ->
> > @Retired
> > > > can
> > > > > > only be done between major releases, because @Public APIs can
> only
> > be
> > > > > > changed between major releases.
> > > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > > - in 2.3, DataStream will be removed.
> > > > > >
> > > > > > Becket mentioned previously (please correct me if I didn't
> > understand
> > > > it
> > > > > > correctly) that users might not check the changes of annotation
> and
> > > the
> > > > > > upgrade process might not be smooth. I was wondering, if users
> > don't
> > > > pay
> > > > > > attention to that, they will also not pay attention to
> @deprecated.
> > > The
> > > > > > migration will not be smooth too even when we stick to the Java
> > > > > > standard @deprecated. Let's go throw it with examples:
> > > > > >
> > > > > > Scenario 1: with @Retired
> > > > > >
> > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should
> > be
> > > > done
> > > > > > for users who still want to use DataStream. The migration is
> > smooth.
> > > > But
> > > > > > they should be aware of the annotation changes, i.e. @Public ->
> > > > @Retired
> > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed.
> > Since
> > > > it
> > > > > is
> > > > > > a major release upgrade, breaking changes are expected.
> > > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed.
> > Since
> > > > > APIs
> > > > > > are marked as @Retired, breaking changes are expected too.
> > > > > >
> > > > > > In addition to fulfilling both rules, there are two more benefits
> > of
> > > > this
> > > > > > option:
> > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > > > 2. no bridge release is required. DataStream API could be
> > deprecated
> > > in
> > > > > 2.x
> > > > > > without breaking any of the two rules. @Retired provides enough
> > > > > flexibility
> > > > > > for future deprecations and upgrades.
> > > > > >
> > > > > > The only issue I can find is that users might be surprised in
> case
> > 3,
> > > > if
> > > > > > they ignore the previous @Public -> @Retired change. But for such
> > > > users,
> > > > > > they will ignore @deprecated annotation too and will have issues
> > with
> > > > > > pure @deprecated
> > > > > > annotation too:
> > > > > >
> > > > > > Scenario 2: with @deprecated
> > > > > >
> > > > > > The example will be slightly different from scenario 1, because
> > > @Public
> > > > > > APIs can only be removed between major releases. As an example,
> > > > > DataStream
> > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and then
> > > > removed
> > > > > in
> > > > > > 2.0
> > > > > >
> > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked
> > as
> > > > > > deprecated are removed between major releases. Rule 1 is ok. The
> 2
> > > > minor
> > > > > > releases period defined in rule 2 sounds like a smooth migration
> > > plan,
> > > > > but
> > > > > > for users in this case, it is still not smooth.
> > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process
> > > with
> > > > > > deprecated APIs. But for those users who ignore annotation
> changes,
> > > > will
> > > > > > have the same issue mentioned in case 1.
> > > > > >
> > > > > > I understood that, compared to scenario 1, users, who ignore
> > @Public
> > > > > > -> @Retired
> > > > > > changes, will not face any breaking changes, after they migrated
> to
> > > > 2.0.
> > > > > > But the cost of it is bridge releases. There might be many bridge
> > > > > releases
> > > > > > in the future. As I mentioned previously, it is not easy to find
> > the
> > > > > right
> > > > > > timing to kick off the deprecation process of APIs before the
> next
> > > > > expected
> > > > > > major release and 2-3 years in advance.
> > > > > >
> > > > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > > > @deprecated
> > > > > > annotation changes
> > > > > >
> > > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in
> > > 1.21
> > > > > >
> > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no
> > smooth
> > > > > > migration.
> > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > > > > since @deprecated is ignored
> > > > > >
> > > > > > In summary, for users who care about the annotation change, they
> > will
> > > > get
> > > > > > more benefits with @Retired. For users who ignore annotation
> > changes,
> > > > > they
> > > > > > will be always facing non-smooth upgrades even using pure
> standard
> > > > > > @deprecated
> > > > > > annotation/process.
> > > > > >
> > > > > > For the case that IDEs support @deprecated, that is true. There
> are
> > > no
> > > > > such
> > > > > > supports for homebuilt annotation. But the intention is to let
> > users
> > > be
> > > > > > aware of it. There will be other ways to do it for users who care
> > > about
> > > > > > annotation changes. For users who do not care, there should be no
> > > > > > difference between with tool support or without. Using @Retired
> > with
> > > > > > @deprecation
> > > > > > together is at least not worse than only using @deprecation
> alone.
> > > And
> > > > > > there are additional benefits with @Retired.
> > > > > >
> > > > > > Just my 2 cents and looking forward to your feedback, especially
> > > > > different
> > > > > > opinions that will help me understand the issue better. Thanks!
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > > <sr...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > > Hi Xintong,
> > > > > > >
> > > > > > > Thanks for the summary, most of the points that you agreed upon
> > > also
> > > > > make
> > > > > > > sense to me.
> > > > > > >
> > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > demoting
> > > > > > APIs
> > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> any
> > > > > version
> > > > > > > bump,
> > > > > > > >   are not good practices.
> > > > > > >
> > > > > > > I hope we can can go beyond calling it “not a good practice"
> and
> > > > reach
> > > > > > > consensus that we will not demote or remove public APIs in
> minor
> > > > > releases
> > > > > > > and (technically) enforce this rule for all contributions. I
> > think
> > > > > it’s a
> > > > > > > reasonable expectation for stability from a project as mature
> as
> > > > Flink.
> > > > > > >
> > > > > > > > I'm personally in favor of option 1. As the migration-period
> > rule
> > > > is
> > > > > > > newly
> > > > > > > > proposed, I think it's fair to make exceptions for cases
> where
> > we
> > > > > > already
> > > > > > > > missed the best chance for planning the deprecation.
> Moreover,
> > I
> > > do
> > > > > > > believe
> > > > > > > > having a quick major version bump does confuse users. Yes, we
> > can
> > > > > > explain
> > > > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> > > other
> > > > > than
> > > > > > > the
> > > > > > > > removal of an deprecated API. But having to explain this
> itself
> > > is
> > > > an
> > > > > > > > indicator that it might be confusing for users.
> > > > > > > >
> > > > > > > >
> > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > understanding,
> > > > > his
> > > > > > > > major point is that a quick major version bump causes barely
> > any
> > > > > actual
> > > > > > > > lose on users, while in option 1 not providing the migration
> > > period
> > > > > or
> > > > > > > > providing a shorter on is an actual lose on users. (@Becket,
> > > please
> > > > > > > correct
> > > > > > > > me if I'm mistaken.)
> > > > > > > >
> > > > > > >
> > > > > > > I would be open to both options and maybe postpone this
> decision
> > > > until
> > > > > we
> > > > > > > have the complete roadmap for Flink 2.0. We would have a better
> > > > > overview
> > > > > > > about the technical consequences, e.g. how hard it would be to
> > > offer
> > > > > and
> > > > > > > maintain both APIs side by side, how well we get both APIs
> > > separated,
> > > > > and
> > > > > > > whether or not we will be able to use the full potential of our
> > > > > breaking
> > > > > > > changes before the DataStream API is completely removed. One
> > > example
> > > > > for
> > > > > > > the last point I can think of would be moving from lazy to
> eager
> > > > state
> > > > > > > declaration. If we believe that it is reasonable to support all
> > > > planned
> > > > > > > changes while maintaining both APIs, I'm leaning towards option
> > 2.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stefan
> > > > > > >
> > > > > > >
> > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <
> tonysong820@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Becket and I had an offline voice call earlier today. We have
> > > > reached
> > > > > > > > consensus on some of the arguments, but still hold different
> > > > opinions
> > > > > > on
> > > > > > > > some others. I'd like to post the outcome here for
> > transparency.
> > > > > > > >
> > > > > > > > We both agree that:
> > > > > > > >
> > > > > > > >   1. Providing a migration period would be beneficial for our
> > > > users,
> > > > > > and
> > > > > > > >   we should do that
> > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > demoting
> > > > > > APIs
> > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> any
> > > > > version
> > > > > > > bump,
> > > > > > > >   are not good practices.
> > > > > > > >   3. Ideally, with this FLIP, developers should be more
> careful
> > > and
> > > > > > plan
> > > > > > > >   API changes ahead. That means:
> > > > > > > >      1. Be more careful with designing APIs and promoting
> them
> > to
> > > > > > Public,
> > > > > > > >      so that they won't be changed / removed very soon, and
> > also
> > > > the
> > > > > > > >      maintenance overhead for keeping them after deprecation
> > > should
> > > > > be
> > > > > > > >      affordable. (I believe this also aligns with John's
> > > opinion.)
> > > > > > > >      2. Plan the deprecation / removal of old APIs that we
> > don't
> > > > want
> > > > > > to
> > > > > > > >      carry to the next major version earlier, so that they'll
> > > have
> > > > > the
> > > > > > > >      2-minor-release migration period before the next major
> > > version
> > > > > > bump.
> > > > > > > >   4. A practical situation is that, we are not ready for
> > > > deprecating
> > > > > > the
> > > > > > > >   DataStream API, nor afford to carry it for another couple
> of
> > > > years.
> > > > > > > >      1. We cannot deprecate it now, because the new
> > > ProcessFunction
> > > > > API
> > > > > > > >      (the planned replacement for DataStream API) is not yet
> > > ready.
> > > > > We
> > > > > > > haven't
> > > > > > > >      planned the ProcessFunction API earlier because the
> > original
> > > > > > > > plan was to do
> > > > > > > >      in-place changes directly on the DataStream API in
> release
> > > > 2.0,
> > > > > > > until the
> > > > > > > >      smooth migration period is proposed. Before this, the
> only
> > > > > > > guarantees we
> > > > > > > >      provide for Public APIs is that they'll stay compatible
> > > until
> > > > > the
> > > > > > > next
> > > > > > > >      major version, which led to the understanding the
> in-place
> > > > > changes
> > > > > > > on
> > > > > > > >      Public APIs should be fine.
> > > > > > > >      2. We also cannot afford carrying the DataStream API for
> > > > another
> > > > > > > >      couple of years. One of the reasons that we want to
> > > refactor /
> > > > > > > replace
> > > > > > > >      DataStream API is that it exposes and depends on too
> many
> > > > > > > > Flink's internal
> > > > > > > >      runtime implementations, which has already limited /
> > blocked
> > > > > plans
> > > > > > > for
> > > > > > > >      Flink's internal improvements for many times. Also due
> to
> > > the
> > > > > wide
> > > > > > > and
> > > > > > > >      transitive dependencies on the internals, maintaining
> two
> > > sets
> > > > > of
> > > > > > > lower
> > > > > > > >      APIs itself is expensive and would be better to keep it
> > > short.
> > > > > > This
> > > > > > > is
> > > > > > > >      admittedly a historical design flaw which should be
> > avoided
> > > in
> > > > > > > future.
> > > > > > > >
> > > > > > > > What we haven't reached consensus is about how to deal with
> the
> > > > > current
> > > > > > > > situation around the DataStream API. There are two options.
> > > > > > > >
> > > > > > > >
> > > > > > > >   1. We say this FLIP is enforced starting release 2.0. For
> > > current
> > > > > 1.x
> > > > > > > >   APIs, we provide a migration period with best effort, while
> > > > > allowing
> > > > > > > >   exceptions for immediate removal in 2.0. That means we will
> > > still
> > > > > try
> > > > > > > with
> > > > > > > >   best effort to get the ProcessFuncion API ready and
> deprecate
> > > the
> > > > > > > >   DataStream API in 1.x, but will also be allowed to remove
> > > > > DataStream
> > > > > > > API in
> > > > > > > >   2.0 if it's not deprecated 2 minor releases before the
> major
> > > > > version
> > > > > > > bump.
> > > > > > > >   2. We strictly follow the process in this FLIP, and will
> > > quickly
> > > > > bump
> > > > > > > >   the major version from 2.x to 3.0 once the migration period
> > for
> > > > > > > DataStream
> > > > > > > >   API is reached.
> > > > > > > >
> > > > > > > > I'm personally in favor of option 1. As the migration-period
> > rule
> > > > is
> > > > > > > newly
> > > > > > > > proposed, I think it's fair to make exceptions for cases
> where
> > we
> > > > > > already
> > > > > > > > missed the best chance for planning the deprecation.
> Moreover,
> > I
> > > do
> > > > > > > believe
> > > > > > > > having a quick major version bump does confuse users. Yes, we
> > can
> > > > > > explain
> > > > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> > > other
> > > > > than
> > > > > > > the
> > > > > > > > removal of an deprecated API. But having to explain this
> itself
> > > is
> > > > an
> > > > > > > > indicator that it might be confusing for users.
> > > > > > > >
> > > > > > > >
> > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > understanding,
> > > > > his
> > > > > > > > major point is that a quick major version bump causes barely
> > any
> > > > > actual
> > > > > > > > lose on users, while in option 1 not providing the migration
> > > period
> > > > > or
> > > > > > > > providing a shorter on is an actual lose on users. (@Becket,
> > > please
> > > > > > > correct
> > > > > > > > me if I'm mistaken.)
> > > > > > > >
> > > > > > > >
> > > > > > > > And we'd like to hear more feedback from the community.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > > vvcephei@apache.org
> > > > > > > <ma...@apache.org>> wrote:
> > > > > > > >
> > > > > > > >> Hi Becket and Xintong,
> > > > > > > >>
> > > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > > >>
> > > > > > > >> Once an API is marked Public, we're committing to support it
> > > until
> > > > > > it's
> > > > > > > >> deprecated, and once it's deprecated, to leave it in place
> for
> > > at
> > > > > > least
> > > > > > > two
> > > > > > > >> minor releases and then only remove it in the following
> major
> > > > > release.
> > > > > > > >>
> > > > > > > >> As a user, I would be dismayed to discover that the project
> > > > > > maintainers
> > > > > > > >> demoted a Public API to PublicEvolving or Experimental in
> > order
> > > to
> > > > > > > violate
> > > > > > > >> the spirit of the deprecation period for Public APIs.
> > Likewise,
> > > > I'd
> > > > > > > view a
> > > > > > > >> rapid sequence of minor releases for the purpose of dropping
> > > > > > deprecated
> > > > > > > >> APIs in the next major release as another violation of the
> > > spirit
> > > > of
> > > > > > our
> > > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > > >>
> > > > > > > >> I'm glad we're gaming out these situations before we
> institute
> > > > this
> > > > > > > FLIP,
> > > > > > > >> and I hope we all converge on a clear understanding of what
> > > we're
> > > > > > > >> guaranteeing. From where I'm sitting, the essence of this
> FLIP
> > > is
> > > > > > > >> specifically to prevent us from doing whatever we want and
> > > instead
> > > > > to
> > > > > > > >> commit to providing a suitable notice period to users who
> rely
> > > on
> > > > > > Public
> > > > > > > >> APIs.
> > > > > > > >>
> > > > > > > >> It honestly shouldn't matter what the maintenance overhead
> is
> > > (or,
> > > > > > > rather,
> > > > > > > >> evaluating the maintenance overhead should be a
> consideration
> > > > before
> > > > > > we
> > > > > > > >> promote an API to Public to begin with). My experience in
> > other
> > > > > > > projects is
> > > > > > > >> that regretting a feature so much you want to drop it asap
> is
> > > > > > extremely
> > > > > > > >> rare, and that living with the pain for a little while until
> > the
> > > > > > > >> deprecation period expires will probably make us better
> > > engineers.
> > > > > > > >>
> > > > > > > >> With that all said, one thing that is not a violation is to
> > > > propose
> > > > > a
> > > > > > > new
> > > > > > > >> major release in order to drop especially burdensome
> > deprecated
> > > > APIs
> > > > > > > that
> > > > > > > >> have already enjoyed their full deprecation period. Those
> > really
> > > > bad
> > > > > > > "case
> > > > > > > >> A" features will be very rare.
> > > > > > > >>
> > > > > > > >> Looking over the FLIP again, I see a reference to the
> release
> > > > > > > >> documentation, but not to the build tooling support for the
> > > > > Deprecated
> > > > > > > >> annotation, so it might help to share this scenario: The way
> > > this
> > > > > will
> > > > > > > all
> > > > > > > >> come together in practice is that, by giving good stability
> > > > > > guarantees,
> > > > > > > we
> > > > > > > >> will encourage our users to bump minor versions when they
> are
> > > > > > released,
> > > > > > > >> which means that they will see compiler warnings immediately
> > if
> > > > any
> > > > > of
> > > > > > > the
> > > > > > > >> APIs they've used are deprecated. In fact, I'd recommend for
> > > them
> > > > to
> > > > > > use
> > > > > > > >> the `-Werror` javac flag to be sure that they don't miss
> those
> > > > > > warnings.
> > > > > > > >> Given our release cadence, this means that they will have
> > about
> > > > six
> > > > > > > months
> > > > > > > >> to migrate off of any deprecated APIs without being forced
> to
> > > > avoid
> > > > > > > >> updates. And they really need to be able to adopt further
> > minor
> > > > > > updates
> > > > > > > in
> > > > > > > >> the series, since they may contain further improvements that
> > > > > actually
> > > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > > >>
> > > > > > > >> The compiler warnings are a far more reliable mechanism to
> > > > advertise
> > > > > > > >> deprecation than release notes, since users' own builds will
> > > > notify
> > > > > > them
> > > > > > > >> right away about only the deprecations that are relevant to
> > > them,
> > > > > but
> > > > > > it
> > > > > > > >> does rely on users feeling ok to bump up minor versions in a
> > > > timely
> > > > > > > >> fashion. If we consistently burn them by not observing the
> > > > stability
> > > > > > > >> guarantees, they'll try to avoid updates instead, and the
> > whole
> > > > > thing
> > > > > > > falls
> > > > > > > >> apart.
> > > > > > > >>
> > > > > > > >> Thanks again,
> > > > > > > >> -John
> > > > > > > >>
> > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > > >>> Hi Xintong,
> > > > > > > >>>
> > > > > > > >>> Let's compare the following cases:
> > > > > > > >>>
> > > > > > > >>> A. If the maintenance overhead of the deprecated API is
> high,
> > > and
> > > > > we
> > > > > > > want
> > > > > > > >>> to remove it after two minor releases. Then there are two
> > > > options:
> > > > > > > >>>    A1: Demote the API in the major version bump and remove
> > the
> > > > code
> > > > > > > >> with a
> > > > > > > >>> minor version bump.
> > > > > > > >>>    A2: Do exactly the same as 1-A, except that when
> removing
> > > the
> > > > > > code,
> > > > > > > >>> bump up the major version. So this might be a short major
> > > > version.
> > > > > > > >>>
> > > > > > > >>> B. If the maintenance overhead of the deprecated API is not
> > > high,
> > > > > > > >> therefore
> > > > > > > >>> keeping it for a long time is affordable. There are also
> two
> > > > > options:
> > > > > > > >>>    B1: Same as A-1, demote the API in the major version
> bump
> > > and
> > > > > > remove
> > > > > > > >>> the code with a minor version bump.
> > > > > > > >>>    B2: Keep the API for all the minor versions in the major
> > > > > version,
> > > > > > > and
> > > > > > > >>> only remove the code in the next major version.
> > > > > > > >>>
> > > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > > >>>
> > > > > > > >>> For case A:
> > > > > > > >>> The following stays the same for A1 and A2:
> > > > > > > >>>  - users will lose the API after two minor leases. So the
> > > > migration
> > > > > > > >> period
> > > > > > > >>> is the same for A-1 and A-2.
> > > > > > > >>>  - A1 and A2 will have exactly the same code after the
> > removal
> > > of
> > > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > > >>>  - To move forward, users need to move to the next minor
> > > version
> > > > in
> > > > > > A1,
> > > > > > > >> or
> > > > > > > >>> to the next major version in A2. Because the code are the
> > same,
> > > > the
> > > > > > > >> actual
> > > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > > >>>
> > > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > > >>>  - A1 allows keeping the major version release cadence. A-2
> > > will
> > > > > > have a
> > > > > > > >>> short major version release.
> > > > > > > >>>  - A1 breaks the well understood API semantic, while A-2
> does
> > > > not.
> > > > > > > >>>
> > > > > > > >>> From what I see, since there is no well establish standard
> > > > > regarding
> > > > > > > how
> > > > > > > >>> long a major version should be released, A short major
> > version
> > > > > > release
> > > > > > > is
> > > > > > > >>> potential and emotional. It is not ideal but does not have
> > > > material
> > > > > > > >>> downsides compared with A1. I did not hear anyone
> complaining
> > > > about
> > > > > > > Kafka
> > > > > > > >>> only has two 1.x release. However, A1 actually breaks the
> > well
> > > > > > > understood
> > > > > > > >>> API semantic, which has more material impact.
> > > > > > > >>>
> > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should
> fall
> > > into
> > > > > > case
> > > > > > > B.
> > > > > > > >>> So, short major versions should be very occasional. I'd be
> > very
> > > > > > > concerned
> > > > > > > >>> if the reason we choose A1 is simply because we cannot
> afford
> > > > > > > >> maintaining a
> > > > > > > >>> bunch of deprecated APIs until the next major version. This
> > > > > indicates
> > > > > > > >> that
> > > > > > > >>> the actual problem we need to solve is to lower the
> > maintenance
> > > > > > > overhead
> > > > > > > >> of
> > > > > > > >>> deprecated APIs, so that we are comfortable to keep them
> > > longer.
> > > > As
> > > > > > > John
> > > > > > > >>> and I mentioned earlier, there are ways to achieve this and
> > we
> > > > need
> > > > > > to
> > > > > > > >>> learn how to do it in Flink. Otherwise, our discussion
> about
> > > > > > versioning
> > > > > > > >>> here does not bring much value, because we will end up
> with a
> > > > bunch
> > > > > > of
> > > > > > > >>> short-lived APIs which upset our users, no matter how we
> > > version
> > > > > the
> > > > > > > >>> releases.
> > > > > > > >>>
> > > > > > > >>> So, if there are concrete examples that you think will
> block
> > us
> > > > > from
> > > > > > > >>> keeping API stability with affordable cost, let's take a
> look
> > > > > > together
> > > > > > > >> and
> > > > > > > >>> see if that can be improved.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>>
> > > > > > > >>> Jiangjie (Becket) Qin
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> From what I see,
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > > tonysong820@gmail.com
> > > > > > > <ma...@gmail.com>>
> > > > > > > >> wrote:
> > > > > > > >>>
> > > > > > > >>>>>
> > > > > > > >>>>> The part I don't understand is if we are willing to have
> a
> > > > > > migration
> > > > > > > >>>>> period, and do a minor version bump to remove an API,
> what
> > do
> > > > we
> > > > > > > >> lose to
> > > > > > > >>>> do
> > > > > > > >>>>> a major version bump instead, so we don't break the
> common
> > > > > > versioning
> > > > > > > >>>>> semantic?
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>> I think we are talking about the cases where a major
> version
> > > > bump
> > > > > > > >> happens
> > > > > > > >>>> before a deprecated Public API reaches its migration
> period.
> > > So
> > > > > > > >> removing
> > > > > > > >>>> the API with another major version bump means we have two
> > > > > > consecutive
> > > > > > > >> major
> > > > > > > >>>> versions in a very short time. And as previously
> mentioned,
> > > > having
> > > > > > > >> frequent
> > > > > > > >>>> major version bumps would weaken the value of the
> commitment
> > > > > "Public
> > > > > > > >> API
> > > > > > > >>>> stay compatible within a major version". I think users
> also
> > > have
> > > > > > > >>>> expectations about how long a major version should live,
> > which
> > > > > > should
> > > > > > > >> be at
> > > > > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > > > > >>>>
> > > > > > > >>>> This is another option, but I think it is very likely more
> > > > > expensive
> > > > > > > >> than
> > > > > > > >>>>> simply bumping the major version. And I imagine there
> will
> > be
> > > > > > > >> questions
> > > > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> > > 2.0"?
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>> Yes, it's more expensive than simply bumping the major
> > > version,
> > > > > but
> > > > > > > >> IMHO
> > > > > > > >>>> it's probably cheaper than carrying the API until the next
> > > major
> > > > > > > >> version if
> > > > > > > >>>> we don't bump the major version very soon. And see my
> reply
> > > > above
> > > > > on
> > > > > > > >> why
> > > > > > > >>>> not bumping immediately.
> > > > > > > >>>>
> > > > > > > >>>> Best,
> > > > > > > >>>>
> > > > > > > >>>> Xintong
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > > becket.qin@gmail.com
> > > > > > > <ma...@gmail.com>>
> > > > > > > >> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Hi Xintong,
> > > > > > > >>>>>
> > > > > > > >>>>> Please see the replies below.
> > > > > > > >>>>>
> > > > > > > >>>>> I see your point, that users would feel surprised if they
> > > find
> > > > > > > >> things no
> > > > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > > > However,
> > > > > > > >> I'd
> > > > > > > >>>>> like
> > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > similar
> > > > > > > >> problem
> > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > attention
> > > > and
> > > > > > > >> make
> > > > > > > >>>>> sure
> > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer
> > be
> > > > > Public
> > > > > > > >> in
> > > > > > > >>>>> 2.0.
> > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > warnings
> > > > > in
> > > > > > > >>>> logs,
> > > > > > > >>>>>> etc.
> > > > > > > >>>>>
> > > > > > > >>>>> First of all, I am not a fan of removing PublicEvolving
> > APIs
> > > in
> > > > > > minor
> > > > > > > >>>>> version changes. Personally speaking, I want them to be
> > also
> > > > > > removed
> > > > > > > >> in
> > > > > > > >>>>> major version changes.
> > > > > > > >>>>>
> > > > > > > >>>>> Empirically, I rarely see projects with complex API
> > semantic
> > > > work
> > > > > > > >> well.
> > > > > > > >>>>> Many, if not most, users don't read docs / release notes
> /
> > > > > warnings
> > > > > > > >> in
> > > > > > > >>>>> logs. I don't think that is their fault. Typically an
> > > > application
> > > > > > > >>>> developer
> > > > > > > >>>>> will have to deal with dozens of libraries maintained by
> > > > various
> > > > > > > >>>>> communities / groups. It is just too difficult for them
> to
> > > keep
> > > > > > > >> track of
> > > > > > > >>>>> all the specific semantics each project puts there. If
> you
> > > > think
> > > > > > > >> about
> > > > > > > >>>> it,
> > > > > > > >>>>> how many of Flink developers read all the release notes
> of
> > > > Guava,
> > > > > > > >> Apache
> > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a
> > > version
> > > > > of
> > > > > > > >> them?
> > > > > > > >>>>> One would probably try bumping the dependency version and
> > see
> > > > if
> > > > > > > >> there is
> > > > > > > >>>>> an exception and solve them case by case. And if it is a
> > > minor
> > > > > > > >> version
> > > > > > > >>>>> bump, one probably does not expect an exception at all.
> > > Another
> > > > > > > >> example
> > > > > > > >>>> is
> > > > > > > >>>>> that in Flink we still have so many usage of deprecated
> > > methods
> > > > > all
> > > > > > > >> over
> > > > > > > >>>>> the place, and I strongly doubt everyone knows when the
> > > source
> > > > > code
> > > > > > > >> of
> > > > > > > >>>>> these methods are deprecated and when they should be
> > removed.
> > > > > > > >>>>>
> > > > > > > >>>>> So, the most important principle of API is simple and
> > > > intuitive.
> > > > > > The
> > > > > > > >>>>> versioning semantic is a simple and universally accepted
> > API
> > > > > > > >> stability
> > > > > > > >>>>> standard. If we ourselves as Flink developers are relying
> > on
> > > > this
> > > > > > > >> for our
> > > > > > > >>>>> own dependencies. I don't think we can expect more from
> our
> > > > > users.
> > > > > > > >>>>>
> > > > > > > >>>>> Another possible alternative: whenever there's a
> deprecated
> > > > > Public
> > > > > > > >> API
> > > > > > > >>>> that
> > > > > > > >>>>>> reaches a major version bump before the migration
> period,
> > > and
> > > > we
> > > > > > > >> also
> > > > > > > >>>>> don't
> > > > > > > >>>>>> want to carry it for all the next major release series,
> we
> > > may
> > > > > > > >> consider
> > > > > > > >>>>>> releasing more minor releases for the previous major
> > version
> > > > > after
> > > > > > > >> the
> > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> then
> > we
> > > > > bump
> > > > > > > >> to
> > > > > > > >>>> 2.0,
> > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > provide
> > > > > users
> > > > > > > >>>> another
> > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> the
> > > > > > > >>>> 2-minor-release
> > > > > > > >>>>>> migration period.
> > > > > > > >>>>>
> > > > > > > >>>>> This is another option, but I think it is very likely
> more
> > > > > > expensive
> > > > > > > >> than
> > > > > > > >>>>> simply bumping the major version. And I imagine there
> will
> > be
> > > > > > > >> questions
> > > > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> > > 2.0"?
> > > > > > > >>>>>
> > > > > > > >>>>> I think my major point is, we should not carry APIs
> > > deprecated
> > > > > in a
> > > > > > > >>>>>> previous major version along all the next major version
> > > > series.
> > > > > > I'd
> > > > > > > >>>> like
> > > > > > > >>>>> to
> > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > period,
> > > > as
> > > > > > > >> long
> > > > > > > >>>> as
> > > > > > > >>>>> it
> > > > > > > >>>>>> does not prevent us from making breaking changes. If it
> > > > doesn't
> > > > > > > >> work,
> > > > > > > >>>> I'd
> > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > fallback
> > > > > to
> > > > > > > >> only
> > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > >>>>>
> > > > > > > >>>>> The part I don't understand is if we are willing to have
> a
> > > > > > migration
> > > > > > > >>>>> period, and do a minor version bump to remove an API,
> what
> > do
> > > > we
> > > > > > > >> lose to
> > > > > > > >>>> do
> > > > > > > >>>>> a major version bump instead, so we don't break the
> common
> > > > > > versioning
> > > > > > > >>>>> semantic?
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks,
> > > > > > > >>>>>
> > > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > > tonysong820@gmail.com
> > > > > > > <ma...@gmail.com>>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> As an end user who only uses Public APIs, if I don't
> > change
> > > > my
> > > > > > > >> code
> > > > > > > >>>> at
> > > > > > > >>>>>>> all, my expectation is the following:
> > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I
> > am
> > > > fine
> > > > > > > >> with
> > > > > > > >>>>> all
> > > > > > > >>>>>>> the 2.x versions.
> > > > > > > >>>>>>> Actually I think there are some dependency version
> > > resolution
> > > > > > > >>>> policies
> > > > > > > >>>>>> out
> > > > > > > >>>>>>> there which picks the highest minor version when the
> > > > > dependencies
> > > > > > > >>>> pull
> > > > > > > >>>>> in
> > > > > > > >>>>>>> multiple minor versions of the same jar, which may be
> > > broken
> > > > if
> > > > > > > >> we
> > > > > > > >>>>> remove
> > > > > > > >>>>>>> the API in minor releases.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> I see your point, that users would feel surprised if
> they
> > > find
> > > > > > > >> things
> > > > > > > >>>> no
> > > > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > > > However,
> > > > > > > >> I'd
> > > > > > > >>>>> like
> > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > similar
> > > > > > > >> problem
> > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > attention
> > > > and
> > > > > > > >> make
> > > > > > > >>>>> sure
> > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer
> > be
> > > > > Public
> > > > > > > >> in
> > > > > > > >>>>> 2.0.
> > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > warnings
> > > > > in
> > > > > > > >>>> logs,
> > > > > > > >>>>>> etc.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Another possible alternative: whenever there's a
> > deprecated
> > > > > Public
> > > > > > > >> API
> > > > > > > >>>>> that
> > > > > > > >>>>>> reaches a major version bump before the migration
> period,
> > > and
> > > > we
> > > > > > > >> also
> > > > > > > >>>>> don't
> > > > > > > >>>>>> want to carry it for all the next major release series,
> we
> > > may
> > > > > > > >> consider
> > > > > > > >>>>>> releasing more minor releases for the previous major
> > version
> > > > > after
> > > > > > > >> the
> > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> then
> > we
> > > > > bump
> > > > > > > >> to
> > > > > > > >>>> 2.0,
> > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > provide
> > > > > users
> > > > > > > >>>> another
> > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> the
> > > > > > > >>>> 2-minor-release
> > > > > > > >>>>>> migration period.
> > > > > > > >>>>>>
> > > > > > > >>>>>> I think my major point is, we should not carry APIs
> > > deprecated
> > > > > in
> > > > > > a
> > > > > > > >>>>>> previous major version along all the next major version
> > > > series.
> > > > > > I'd
> > > > > > > >>>> like
> > > > > > > >>>>> to
> > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > period,
> > > > as
> > > > > > > >> long
> > > > > > > >>>> as
> > > > > > > >>>>> it
> > > > > > > >>>>>> does not prevent us from making breaking changes. If it
> > > > doesn't
> > > > > > > >> work,
> > > > > > > >>>> I'd
> > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > fallback
> > > > > to
> > > > > > > >> only
> > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Best,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Xintong
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > > vvcephei@apache.org
> > > > > > > <ma...@apache.org>
> > > > > > > >>>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hi Becket,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> > conversation
> > > > > about
> > > > > > > >>>>>>> compatibility outside of this FLIP thread, but for
> now, I
> > > can
> > > > > > > >> accept
> > > > > > > >>>>> your
> > > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks again,
> > > > > > > >>>>>>> John
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > >>>>>>>> Hi John,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in major
> > > > > releases
> > > > > > > >>>>> across
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> board? I understand that Experimental and
> > PublicEvolving
> > > > APIs
> > > > > > > >> are
> > > > > > > >>>> by
> > > > > > > >>>>>>>>> definition less stable, but it seems like this should
> > be
> > > > > > > >> reflected
> > > > > > > >>>>> in
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must
> > keep
> > > > > them
> > > > > > > >>>>> around
> > > > > > > >>>>>>> for
> > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > drop
> > > > them
> > > > > > > >> in a
> > > > > > > >>>>>>> minor or
> > > > > > > >>>>>>>>> patch release.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Personally speaking, I would love to do this, for
> > exactly
> > > > the
> > > > > > > >>>> reason
> > > > > > > >>>>>> you
> > > > > > > >>>>>>>> mentioned. However, I did not propose this due to the
> > > > > following
> > > > > > > >>>>>> reasons:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> > > accepted
> > > > > > > >> FLIPs
> > > > > > > >>>> too
> > > > > > > >>>>>>> soon.
> > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > > development.
> > > > At
> > > > > > > >> this
> > > > > > > >>>>>>> point,
> > > > > > > >>>>>>>> Flink still lacks some design / routines to support
> good
> > > API
> > > > > > > >>>>>>> evolvability /
> > > > > > > >>>>>>>> extensibility. Just like you said, it takes some time
> to
> > > be
> > > > > > > >> good at
> > > > > > > >>>>>> this.
> > > > > > > >>>>>>>> In this case, my concern is that only removing
> > > Experimental
> > > > /
> > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may
> result
> > in
> > > > too
> > > > > > > >> much
> > > > > > > >>>>>>>> overhead and dramatically slow down the development of
> > > > Flink.
> > > > > > > >> So, I
> > > > > > > >>>>> was
> > > > > > > >>>>>>>> thinking that we can start with the current status.
> > > > Hopefully
> > > > > > > >> after
> > > > > > > >>>>> we
> > > > > > > >>>>>>> are
> > > > > > > >>>>>>>> more comfortable with the maintenance overhead of
> > > deprecated
> > > > > > > >> APIs,
> > > > > > > >>>> we
> > > > > > > >>>>>> can
> > > > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > > > >> PublicEvolving
> > > > > > > >>>>> APIs.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Hi Becket,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is
> > > > really
> > > > > > > >>>>>> important.
> > > > > > > >>>>>>> I
> > > > > > > >>>>>>>>> understand some people’s concerns about the
> additional
> > > > burden
> > > > > > > >> for
> > > > > > > >>>>>>> project
> > > > > > > >>>>>>>>> maintainers, but my personal experience with Kafka
> has
> > > been
> > > > > > > >> that
> > > > > > > >>>>> it’s
> > > > > > > >>>>>>> very
> > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to
> users.
> > > In
> > > > > > > >> fact,
> > > > > > > >>>>> users
> > > > > > > >>>>>>>>> being able to confidently upgrade is also a benefit
> to
> > > > > > > >>>> maintainers,
> > > > > > > >>>>> as
> > > > > > > >>>>>>> we
> > > > > > > >>>>>>>>> will get fewer questions from people stuck on very
> old
> > > > > > > >> versions.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> One question:
> > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in
> major
> > > > > > > >> releases
> > > > > > > >>>>> across
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> board? I understand that Experimental and
> > PublicEvolving
> > > > APIs
> > > > > > > >> are
> > > > > > > >>>> by
> > > > > > > >>>>>>>>> definition less stable, but it seems like this should
> > be
> > > > > > > >> reflected
> > > > > > > >>>>> in
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must
> > keep
> > > > > them
> > > > > > > >>>>> around
> > > > > > > >>>>>>> for
> > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > drop
> > > > them
> > > > > > > >> in a
> > > > > > > >>>>>>> minor or
> > > > > > > >>>>>>>>> patch release.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> The advantage of forbidding the removal of any API in
> > > minor
> > > > > or
> > > > > > > >>>> patch
> > > > > > > >>>>>>>>> releases is that users will get a strong guarantee
> that
> > > > they
> > > > > > > >> can
> > > > > > > >>>>> bump
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>> minor or patch version and still be able to compile,
> or
> > > > even
> > > > > > > >> just
> > > > > > > >>>>>>> re-link
> > > > > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions
> at
> > > run
> > > > > > > >> time.
> > > > > > > >>>>> This
> > > > > > > >>>>>>> is a
> > > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> > Experimental
> > > > > APIs
> > > > > > > >>>>> outside
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>>>> major releases, users can no longer confidently
> > upgrade.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of
> > > > points:
> > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation instead
> of
> > > > > > > >>>> introducing
> > > > > > > >>>>>> our
> > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > > > > integrated
> > > > > > > >>>> into
> > > > > > > >>>>>>> build
> > > > > > > >>>>>>>>> tools and IDEs.
> > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this
> > FLIP;
> > > > it
> > > > > > > >> seems
> > > > > > > >>>>>>>>> orthogonal, and something that should probably be
> taken
> > > > > > > >>>> case-by-case
> > > > > > > >>>>>>>>> anyway.
> > > > > > > >>>>>>>>> * Aside from deprecation and removal, there have been
> > > some
> > > > > > > >>>>> discussions
> > > > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible
> > ways.
> > > > > > > >> This is
> > > > > > > >>>>>>> somewhat
> > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it
> > before,
> > > > > it’ll
> > > > > > > >>>> take
> > > > > > > >>>>>> some
> > > > > > > >>>>>>>>> time to become good at it. I feel like this topic
> > should
> > > > also
> > > > > > > >> be
> > > > > > > >>>>>>> orthogonal
> > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to
> > adopt a
> > > > > > > >> simple
> > > > > > > >>>>>> policy
> > > > > > > >>>>>>> not
> > > > > > > >>>>>>>>> to break existing user programs, and leave the “how”
> up
> > > to
> > > > > > > >>>>>> implementers
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>> reviewers.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks again,
> > > > > > > >>>>>>>>> John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > >>>>>>>>>> Hi All,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong
> is
> > a
> > > > > > > >> great
> > > > > > > >>>>> idea.
> > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > > > > >>>> @PublicEvolving
> > > > > > > >>>>>> --(2
> > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > > > > > > >> flexibility
> > > > > > > >>>>>>> without
> > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed
> to
> > > > > > > >> change
> > > > > > > >>>>>> between
> > > > > > > >>>>>>>>> major
> > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> > > provides
> > > > > > > >>>>> additional
> > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> > > themself
> > > > > > > >> are
> > > > > > > >>>> not
> > > > > > > >>>>>>>>> changed.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> > issues. I
> > > > > > > >> want to
> > > > > > > >>>>>> move
> > > > > > > >>>>>>> one
> > > > > > > >>>>>>>>>> step further and suggest introducing one more
> > annotation
> > > > > > > >>>> @Retired.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > > > > > > >> downgrading
> > > > > > > >>>>>>> @Public
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply,
> > > Java
> > > > > > > >>>> standard
> > > > > > > >>>>>>>>>> @deprecated should be used in the early stage of the
> > > > > > > >> deprecation
> > > > > > > >>>>>>> process
> > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java
> > does
> > > > not
> > > > > > > >>>> allow
> > > > > > > >>>>>> us
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>> extend annotation, I think it would be feasible to
> > have
> > > > the
> > > > > > > >> new
> > > > > > > >>>>>>> @Retired
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>> help us monitor and manage the deprecation process,
> > > house
> > > > > > > >>>>> cleaning,
> > > > > > > >>>>>>> etc.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> @Retired:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > > > > >> compatibility
> > > > > > > >>>>>> before
> > > > > > > >>>>>>>>> APIs
> > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> > > processes
> > > > > > > >> on
> > > > > > > >>>> the
> > > > > > > >>>>> ML
> > > > > > > >>>>>>> are
> > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1
> > minor
> > > > > > > >> release
> > > > > > > >>>>>>> sharply
> > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check
> > whether
> > > > it
> > > > > > > >> is
> > > > > > > >>>>>>> possible).
> > > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to
> > > remove
> > > > > > > >> them.
> > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> > > > @Retired
> > > > > > > >> are
> > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal
> > ->
> > > > > > > >> @Retired
> > > > > > > >>>>>> could
> > > > > > > >>>>>>>>> also
> > > > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > > > >>>> dev-friendliness,
> > > > > > > >>>>>> but
> > > > > > > >>>>>>> not
> > > > > > > >>>>>>>>>> mandatory.
> > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to
> > support
> > > > the
> > > > > > > >>>>>>> deprecation
> > > > > > > >>>>>>>>>> process management. Further extension is possible,
> > since
> > > > the
> > > > > > > >>>>>>> annotation
> > > > > > > >>>>>>>>> is
> > > > > > > >>>>>>>>>> built by us.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Best regards,
> > > > > > > >>>>>>>>>> Jing
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > > > >>>>>>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies
> > > inline
> > > > > > > >>>> below.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a
> Public
> > > API
> > > > > > > >> is
> > > > > > > >>>>> also a
> > > > > > > >>>>>>>>> kind of
> > > > > > > >>>>>>>>>>>> such change, just like removing one, which can
> only
> > > > > > > >> happen
> > > > > > > >>>> with
> > > > > > > >>>>>>> major
> > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > > > Public
> > > > > > > >>>> APIs
> > > > > > > >>>>>>>>> anytime,
> > > > > > > >>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > > > >> reaching
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> demoting
> > > > > > > >> would
> > > > > > > >>>> be a
> > > > > > > >>>>>>> weaker
> > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> upon
> > > > > > > >> major
> > > > > > > >>>>>> version
> > > > > > > >>>>>>>>> bumps,
> > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > 2-minor-release
> > > > > > > >>>>>> migration
> > > > > > > >>>>>>>>>>> period.
> > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > `@PublicEvolving`
> > > > > > > >> sounds
> > > > > > > >>>>>>> against
> > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > > > >> necessary,
> > > > > > > >>>>>> e.g.,
> > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I
> don't
> > > > > > > >> change my
> > > > > > > >>>>> code
> > > > > > > >>>>>>> at
> > > > > > > >>>>>>>>> all,
> > > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > issue, I
> > > > am
> > > > > > > >> fine
> > > > > > > >>>>>> with
> > > > > > > >>>>>>> all
> > > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > > >>>>>>>>>>> Actually I think there are some dependency version
> > > > > > > >> resolution
> > > > > > > >>>>>>> policies
> > > > > > > >>>>>>>>> out
> > > > > > > >>>>>>>>>>> there which picks the highest minor version when
> the
> > > > > > > >>>> dependencies
> > > > > > > >>>>>>> pull
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which may
> be
> > > > > > > >> broken if
> > > > > > > >>>>> we
> > > > > > > >>>>>>>>> remove
> > > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely
> "legal"
> > > > that
> > > > > > > >> we
> > > > > > > >>>>> bump
> > > > > > > >>>>>> up
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> needed.
> > > > > > > >> However,
> > > > > > > >>>>> this
> > > > > > > >>>>>>> also
> > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> APIs
> > > > will
> > > > > > > >>>> stay
> > > > > > > >>>>>>> stable
> > > > > > > >>>>>>>>>>>> within the major release series, as the series can
> > end
> > > > > > > >>>> anytime.
> > > > > > > >>>>>>> IMHO,
> > > > > > > >>>>>>>>>>> short
> > > > > > > >>>>>>>>>>>> major release series are not something "make the
> end
> > > > > > > >> users
> > > > > > > >>>>>> happy",
> > > > > > > >>>>>>> but
> > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > > > >> frequent
> > > > > > > >>>>>> breaking
> > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > approach,
> > > we
> > > > > > > >> can
> > > > > > > >>>>>> still
> > > > > > > >>>>>>>>> have
> > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > Public
> > > > > > > >> APIs
> > > > > > > >>>>>>>>> deprecated
> > > > > > > >>>>>>>>>>> at
> > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> removed
> > in
> > > > > > > >> the
> > > > > > > >>>> next
> > > > > > > >>>>>>> major
> > > > > > > >>>>>>>>>>>> version.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I totally agree that frequent major version bumps
> are
> > > not
> > > > > > > >>>> ideal,
> > > > > > > >>>>>> but
> > > > > > > >>>>>>>>> here
> > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump which
> > > > > > > >> removes a
> > > > > > > >>>>>> Public
> > > > > > > >>>>>>>>> API.
> > > > > > > >>>>>>>>>>> So the context is that we have already decided to
> > > remove
> > > > > > > >> this
> > > > > > > >>>>>> Public
> > > > > > > >>>>>>> API
> > > > > > > >>>>>>>>>>> while keeping everything else backwards
> compatible. I
> > > > > > > >> think a
> > > > > > > >>>>> major
> > > > > > > >>>>>>>>> version
> > > > > > > >>>>>>>>>>> bump is a commonly understood signal here, compared
> > > with
> > > > a
> > > > > > > >>>> minor
> > > > > > > >>>>>>> version
> > > > > > > >>>>>>>>>>> change. From end users' perspective, for those who
> > are
> > > > not
> > > > > > > >>>>>> impacted,
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>>>> this case upgrading a major version is not
> > necessarily
> > > > more
> > > > > > > >>>>>> involved
> > > > > > > >>>>>>>>> than
> > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as
> smooth
> > > as a
> > > > > > > >>>>>> dependency
> > > > > > > >>>>>>>>>>> version change. For those who are impacted, they
> will
> > > > lose
> > > > > > > >> the
> > > > > > > >>>>>> Public
> > > > > > > >>>>>>>>> API
> > > > > > > >>>>>>>>>>> anyways and a major version bump ensures there is
> no
> > > > > > > >> surprise.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Public API is a well defined common concept, and
> one
> > > of
> > > > > > > >> its
> > > > > > > >>>>>>>>>>>>> convention is that it only changes with a major
> > > version
> > > > > > > >>>>> change.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a
> > Public
> > > > > > > >> API is
> > > > > > > >>>>>> also a
> > > > > > > >>>>>>>>> kind
> > > > > > > >>>>>>>>>>>> of such change, just like removing one, which can
> > only
> > > > > > > >> happen
> > > > > > > >>>>>> with
> > > > > > > >>>>>>>>> major
> > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > > > Public
> > > > > > > >>>> APIs
> > > > > > > >>>>>>>>> anytime,
> > > > > > > >>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > > > >> reaching
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> demoting
> > > > > > > >> would
> > > > > > > >>>> be a
> > > > > > > >>>>>>> weaker
> > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> upon
> > > > > > > >> major
> > > > > > > >>>>>> version
> > > > > > > >>>>>>>>> bumps,
> > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > 2-minor-release
> > > > > > > >>>>>> migration
> > > > > > > >>>>>>>>>>> period.
> > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > `@PublicEvolving`
> > > > > > > >> sounds
> > > > > > > >>>>>>> against
> > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > > > >> necessary,
> > > > > > > >>>>>> e.g.,
> > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> But it should be
> > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if we
> > > really
> > > > > > > >> want
> > > > > > > >>>>> to
> > > > > > > >>>>>>> get
> > > > > > > >>>>>>>>> rid
> > > > > > > >>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > "legal"
> > > > > > > >> that we
> > > > > > > >>>>>> bump
> > > > > > > >>>>>>> up
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> needed.
> > > > > > > >> However,
> > > > > > > >>>>> this
> > > > > > > >>>>>>> also
> > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> APIs
> > > > will
> > > > > > > >>>> stay
> > > > > > > >>>>>>> stable
> > > > > > > >>>>>>>>>>>> within the major release series, as the series can
> > end
> > > > > > > >>>> anytime.
> > > > > > > >>>>>>> IMHO,
> > > > > > > >>>>>>>>>>> short
> > > > > > > >>>>>>>>>>>> major release series are not something "make the
> end
> > > > > > > >> users
> > > > > > > >>>>>> happy",
> > > > > > > >>>>>>> but
> > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > > > >> frequent
> > > > > > > >>>>>> breaking
> > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > approach,
> > > we
> > > > > > > >> can
> > > > > > > >>>>>> still
> > > > > > > >>>>>>>>> have
> > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > Public
> > > > > > > >> APIs
> > > > > > > >>>>>>>>> deprecated
> > > > > > > >>>>>>>>>>> at
> > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> removed
> > in
> > > > > > > >> the
> > > > > > > >>>> next
> > > > > > > >>>>>>> major
> > > > > > > >>>>>>>>>>>> version.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Given our track record I would prefer a regular
> > cycle
> > > > > > > >> (1-2
> > > > > > > >>>>> years)
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not
> > put
> > > > it
> > > > > > > >>>> again
> > > > > > > >>>>>> to
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > expectation
> > > > on
> > > > > > > >>>> when
> > > > > > > >>>>>>>>> breaking
> > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good
> > time
> > > > > > > >> for new
> > > > > > > >>>>>> major
> > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > > > > > > >> needed.
> > > > > > > >>>> That
> > > > > > > >>>>>>> makes
> > > > > > > >>>>>>>>> 1-2
> > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while
> > > leaving
> > > > > > > >> us
> > > > > > > >>>> more
> > > > > > > >>>>>>> time
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Xintong
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler
> <
> > > > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing
> > minor
> > > > > > > >>>>> versions
> > > > > > > >>>>>>> for
> > > > > > > >>>>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure
> how
> > > > > > > >> viable
> > > > > > > >>>> that
> > > > > > > >>>>>> is
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>>>>>> practice.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > > > > > > >> support
> > > > > > > >>>> for
> > > > > > > >>>>>> Java
> > > > > > > >>>>>>>>> 8/11,
> > > > > > > >>>>>>>>>>>>> which may very well be a problem for our current
> > > users.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> > confusion
> > > > > > > >> about
> > > > > > > >>>>>> Public
> > > > > > > >>>>>>>>> API
> > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a
> > mistake
> > > > > > > >> in
> > > > > > > >>>> the
> > > > > > > >>>>>>> NOTES
> > > > > > > >>>>>>>>> part
> > > > > > > >>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just
> > fixed
> > > > > > > >> it.
> > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that
> > we
> > > > > > > >> are on
> > > > > > > >>>>> the
> > > > > > > >>>>>>> same
> > > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > > > > > > >> really
> > > > > > > >>>> want
> > > > > > > >>>>>> to
> > > > > > > >>>>>>> get
> > > > > > > >>>>>>>>>>> rid
> > > > > > > >>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to
> > get
> > > us
> > > > > > > >> to
> > > > > > > >>>>> 2.0.
> > > > > > > >>>>>> ;)
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the
> > > cadence
> > > > > > > >> of
> > > > > > > >>>>> major
> > > > > > > >>>>>>>>> releases
> > > > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> > > opinions
> > > > > > > >> on
> > > > > > > >>>>> that.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases
> > > > between
> > > > > > > >>>> major
> > > > > > > >>>>>>> ones,
> > > > > > > >>>>>>>>>>>>> that for us means that users have to potentially
> > deal
> > > > > > > >> with
> > > > > > > >>>>>>> breaking
> > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular
> > cycle
> > > > > > > >> (1-2
> > > > > > > >>>>>> years)
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not
> > put
> > > > it
> > > > > > > >>>> again
> > > > > > > >>>>>> to
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > expectation
> > > > on
> > > > > > > >>>> when
> > > > > > > >>>>>>>>> breaking
> > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate
> > thread.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in
> > mind?
> > > > > > > >> Not
> > > > > > > >>>>> sure
> > > > > > > >>>>>> I
> > > > > > > >>>>>>>>> fully
> > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change
> here.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> > performance
> > > in
> > > > > > > >>>>> certain
> > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> > unknowingly)
> > > > > > > >> relied
> > > > > > > >>>>> upon
> > > > > > > >>>>>>>>>>>>> (
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > > ),
> > > > > > > a semantic change to some
> > > > > > > >> API.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> > submission.
> > > A
> > > > > > > >> few
> > > > > > > >>>>>>> releases
> > > > > > > >>>>>>>>> back
> > > > > > > >>>>>>>>>>>>> we made changes such that the initialization of
> the
> > > job
> > > > > > > >>>> master
> > > > > > > >>>>>>>>> happens
> > > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > > >>>>>>>>>>>>> This meant the job submission call returns
> sooner,
> > > and
> > > > > > > >> the
> > > > > > > >>>> job
> > > > > > > >>>>>>> state
> > > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change,
> but
> > > the
> > > > > > > >>>>> observed
> > > > > > > >>>>>>>>> behavior
> > > > > > > >>>>>>>>>>>>> may be different.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over time
> we
> > > > > > > >> changed
> > > > > > > >>>>> what
> > > > > > > >>>>>>> some
> > > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi all,

Just want to make sure we are on the same page. There is another example[1]
I was aware of recently that shows why more factors need to be taken care
of than just the migration period. Thanks Galen for your hint.

To put it simply, the concern about API deprecation is not that deprecated
APIs have been removed too early (min migration period is required). The
major concern is that APIs are marked as deprecated for a (too) long time,
much longer than the migration period discussed in this thread, afaik.
Since there is no clear picture/definition, no one knows when to do the
migration for users(after the migration period has expired) and when to
remove deprecated APIs for Flink developers.

Based on all the information I knew, there are two kinds of obstacles that
will and should block the deprecation process:

1. Lack of functionalities in new APIs. It happens e.g. with the
SourceFunction to FLIP-27 Source migration. Users who rely on those
functions can not migrate to new APIs.
2. new APIs have critical bugs. An example could be found at [1]. Users
have to stick to the deprecated APIs.

Since FLIP-321 is focusing on the API deprecation process, those blocking
issues deserve attention and should be put into the FLIP. The current FLIP
seems to only focus on migration periods. If we consider those blocking
issues as orthogonal issues that are beyond the scope of this discussion,
does it make sense to change the FLIP title to something like "Introduce
minimum migration periods of API deprecation process"?

Best regards,
Jing

[1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc

On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <im...@gmail.com> wrote:

> I agree with Jingsong and Becket.
>
> Look at the legacy SourceFunction (a small part of DataStream API),
> the SourceFunction is still not and can't be marked deprecated[1] until
> now after the new Source was released 2 years ago, because the new Source
> still can't fully consume the abilities of legacy API. Considering
> DataStream
> API is the most fundamental and complex API of Flink, I think it is worth
> a longer time than the general process for the deprecation period to
> wait for the new API be mature. The above 2 options sound a bit of rush
> for such a widely used API.
>
> I fully understand the concern of maintenance overhead, but it's a bit hard
> for others to estimate maintenance costs without a concrete design and code
> of the new ProcessFunction API. I agree with Becket that maybe we can
> re-evaluate the API deprecation process once we have the new
> ProcessFunction
> API. If the maintenance is indeed huge, I think it is reasonable to have a
> special rule for this case at that time.
>
> Best,
> Jark
>
>
> [1]: https://issues.apache.org/jira/browse/FLINK-28045
>
> On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com> wrote:
>
> > Hi Jingsong,
> >
> > Thanks for the reply. I completely agree with you.
> >
> > The above 2 options are based on the assumption that the community cannot
> > afford to maintain the deprecated DataStream API for long. I'd say we
> > should try everything we can to maintain it for as much time as possible.
> > DataStream API is actually the most used API in Flink by so many users at
> > this point. Removing it any time soon will dramatically hurt our users.
> So
> > ideally we should keep it for at least 2 years after deprecation, if not
> > more.
> >
> > The prohibitively high maintenance overhead is just an assumption.
> > Personally speaking, I don't feel this assumption is necessarily true. We
> > should re-evaluate once we have the new ProcessFunction API in place.
> > Without the code it is hard to tell for sure. I am actually kind of
> > optimistic about the maintenance cost.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <ji...@gmail.com>
> > wrote:
> >
> > > Thanks Becket and all for your discussion.
> > >
> > > > 1. We say this FLIP is enforced starting release 2.0. For current 1.x
> > > APIs,
> > > we provide a migration period with best effort, while allowing
> exceptions
> > > for immediate removal in 2.0. That means we will still try with best
> > effort
> > > to get the ProcessFuncion API ready and deprecate the DataStream API in
> > > 1.x, but will also be allowed to remove DataStream API in 2.0 if it's
> not
> > > deprecated 2 minor releases before the major version bump.
> > >
> > > > 2. We strictly follow the process in this FLIP, and will quickly bump
> > the
> > > major version from 2.x to 3.0 once the migration period for DataStream
> > API
> > > is reached.
> > >
> > > Sorry, I didn't read the previous detailed discussion because the
> > > discussion list was so long.
> > >
> > > I don't really like either of these options.
> > >
> > > Considering that DataStream is such an important API, can we offer a
> > third
> > > option:
> > >
> > > 3. Maintain the DataStream API throughout 2.X and remove it until 3.x.
> > But
> > > there's no need to assume that 2.X is a short version, it's still a
> > normal
> > > major version.
> > >
> > > Best,
> > > Jingsong
> > >
> > > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> > >
> > > > Thanks much for the input, John, Stefan and Jing.
> > > >
> > > > I think Xingtong has well summarized the pros and cons of the two
> > > options.
> > > > Let's collect a few more opinions here and we can move forward with
> the
> > > one
> > > > more people prefer.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks Xingtong for the summary. If I could only choose one of the
> > > given
> > > > > two options, I would go with option 1. I understood that option 2
> > > worked
> > > > > great with Kafka. But the bridge release will still confuse users
> and
> > > my
> > > > > gut feeling is that many users will skip 2.0 and be waiting for 3.0
> > or
> > > > even
> > > > > 3.x. And since fewer users will use Flink 2.x, the development
> focus
> > > will
> > > > > be on Flink 3.0 with the fact that the current Flink release is
> 1.17
> > > and
> > > > we
> > > > > are preparing 2.0 release. That is weird for me.
> > > > >
> > > > > THB, I would not name the change from @Public to @Retired as a
> > > demotion.
> > > > > The purpose of @Retire is to extend the API lifecycle with one more
> > > > stage,
> > > > > like in the real world, people born, studied, graduated, worked,
> and
> > > > > retired. Afaiu from the previous discussion, there are two rules
> we'd
> > > > like
> > > > > to follow simultaneously:
> > > > >
> > > > > 1. Public APIs can only be changed between major releases.
> > > > > 2. A smooth migration phase should be offered to users, i.e. at
> > least 2
> > > > > minor releases after APIs are marked as @deprecated. There should
> be
> > > new
> > > > > APIs as the replacement.
> > > > >
> > > > > Agree, those rules are good to improve the user friendliness.
> Issues
> > we
> > > > > discussed are rising because we want to fulfill both of them. If we
> > > take
> > > > > care of deprecation very seriously, APIs can be marked as
> > @Deprecated,
> > > > only
> > > > > when the new APIs as the replacement provide all functionalities
> the
> > > > > deprecated APIs have. In an ideal case without critical bugs that
> > might
> > > > > stop users adopting the new APIs. Otherwise the expected
> > "replacement"
> > > > will
> > > > > not happen. Users will still stick to the deprecated APIs, because
> > the
> > > > new
> > > > > APIs can not be used. For big features, it will need at least 4
> minor
> > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > > >
> > > > > - 1st minor release to build the new APIs as the replacement and
> > > waiting
> > > > > for feedback. It might be difficult to mark the old API as
> deprecated
> > > in
> > > > > this release, because we are not sure if the new APIs could cover
> > 100%
> > > > > functionalities.
> > > > > -  In the lucky case,  mark all old APIs as deprecated in the 2nd
> > minor
> > > > > release. (I would even suggest having the new APIs released at
> least
> > > for
> > > > > two minor releases before marking it as deprecated to make sure
> they
> > > can
> > > > > really replace the old APIs, in case we care more about smooth
> > > migration)
> > > > > - 3rd minor release for the migration period
> > > > > -  In another lucky case, the 4th release is a major release, the
> > > > > deprecated APIs could be removed.
> > > > >
> > > > > The above described scenario works only in an ideal case. In
> reality,
> > > it
> > > > > might take longer to get the new APIs ready and mark the old API
> > > > > deprecated. Furthermore, if the 4th release is not a major release,
> > we
> > > > will
> > > > > have to maintain both APIs for many further minor releases. The
> > > question
> > > > is
> > > > > how to know the next major release in advance, especially 4 minor
> > > > releases'
> > > > > period, i.e. more than 2 years in advance? Given that Flink
> contains
> > > many
> > > > > modules, it is difficult to ask devs to create a 2-3 years
> > deprecation
> > > > plan
> > > > > for each case. In case we want to build major releases at a fast
> > pace,
> > > > > let's say every two years, it means devs must plan any API
> > deprecation
> > > > > right after each major release. Afaiac, it is quite difficult.
> > > > >
> > > > > The major issue is, afaiu, if we follow rule 2, we have to keep all
> > > > @Public
> > > > > APIs, e.g. DataStream, that are not marked as deprecated yet, to
> 2.0.
> > > > Then
> > > > > we have to follow rule 1 to keep it unchanged until we have 3.0.
> That
> > > is
> > > > > why @Retired is useful to give devs more flexibility and still
> > fulfill
> > > > both
> > > > > rules. Let's check it with examples:
> > > > >
> > > > > - we have @Public DataStream API in 1.18. It will not be marked
> > > > > as @Deprecated, because the new APIs as the replacement are not
> > ready.
> > > > > - we keep the DataStream API itself unchanged in 2.0, but change
> the
> > > > > annotation from @Public to @Retire. New APIs will be introduced
> too.
> > In
> > > > > this case, Rule 1 is ok, since public API is allowed to change
> > between
> > > > > major releases. Only changing annotation is the minimal change we
> > could
> > > > do
> > > > > and it does not break rule 1. Rule 2 is ok too, since the
> DataStream
> > > APIs
> > > > > work exactly the same.  Attention: the change of @Public ->
> @Retired
> > > can
> > > > > only be done between major releases, because @Public APIs can only
> be
> > > > > changed between major releases.
> > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > - in 2.3, DataStream will be removed.
> > > > >
> > > > > Becket mentioned previously (please correct me if I didn't
> understand
> > > it
> > > > > correctly) that users might not check the changes of annotation and
> > the
> > > > > upgrade process might not be smooth. I was wondering, if users
> don't
> > > pay
> > > > > attention to that, they will also not pay attention to @deprecated.
> > The
> > > > > migration will not be smooth too even when we stick to the Java
> > > > > standard @deprecated. Let's go throw it with examples:
> > > > >
> > > > > Scenario 1: with @Retired
> > > > >
> > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should
> be
> > > done
> > > > > for users who still want to use DataStream. The migration is
> smooth.
> > > But
> > > > > they should be aware of the annotation changes, i.e. @Public ->
> > > @Retired
> > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed.
> Since
> > > it
> > > > is
> > > > > a major release upgrade, breaking changes are expected.
> > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed.
> Since
> > > > APIs
> > > > > are marked as @Retired, breaking changes are expected too.
> > > > >
> > > > > In addition to fulfilling both rules, there are two more benefits
> of
> > > this
> > > > > option:
> > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > > 2. no bridge release is required. DataStream API could be
> deprecated
> > in
> > > > 2.x
> > > > > without breaking any of the two rules. @Retired provides enough
> > > > flexibility
> > > > > for future deprecations and upgrades.
> > > > >
> > > > > The only issue I can find is that users might be surprised in case
> 3,
> > > if
> > > > > they ignore the previous @Public -> @Retired change. But for such
> > > users,
> > > > > they will ignore @deprecated annotation too and will have issues
> with
> > > > > pure @deprecated
> > > > > annotation too:
> > > > >
> > > > > Scenario 2: with @deprecated
> > > > >
> > > > > The example will be slightly different from scenario 1, because
> > @Public
> > > > > APIs can only be removed between major releases. As an example,
> > > > DataStream
> > > > > API will be marked as deprecated in 1.19, kept in 1.20, and then
> > > removed
> > > > in
> > > > > 2.0
> > > > >
> > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked
> as
> > > > > deprecated are removed between major releases. Rule 1 is ok. The 2
> > > minor
> > > > > releases period defined in rule 2 sounds like a smooth migration
> > plan,
> > > > but
> > > > > for users in this case, it is still not smooth.
> > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process
> > with
> > > > > deprecated APIs. But for those users who ignore annotation changes,
> > > will
> > > > > have the same issue mentioned in case 1.
> > > > >
> > > > > I understood that, compared to scenario 1, users, who ignore
> @Public
> > > > > -> @Retired
> > > > > changes, will not face any breaking changes, after they migrated to
> > > 2.0.
> > > > > But the cost of it is bridge releases. There might be many bridge
> > > > releases
> > > > > in the future. As I mentioned previously, it is not easy to find
> the
> > > > right
> > > > > timing to kick off the deprecation process of APIs before the next
> > > > expected
> > > > > major release and 2-3 years in advance.
> > > > >
> > > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > > @deprecated
> > > > > annotation changes
> > > > >
> > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in
> > 1.21
> > > > >
> > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no
> smooth
> > > > > migration.
> > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > > > since @deprecated is ignored
> > > > >
> > > > > In summary, for users who care about the annotation change, they
> will
> > > get
> > > > > more benefits with @Retired. For users who ignore annotation
> changes,
> > > > they
> > > > > will be always facing non-smooth upgrades even using pure standard
> > > > > @deprecated
> > > > > annotation/process.
> > > > >
> > > > > For the case that IDEs support @deprecated, that is true. There are
> > no
> > > > such
> > > > > supports for homebuilt annotation. But the intention is to let
> users
> > be
> > > > > aware of it. There will be other ways to do it for users who care
> > about
> > > > > annotation changes. For users who do not care, there should be no
> > > > > difference between with tool support or without. Using @Retired
> with
> > > > > @deprecation
> > > > > together is at least not worse than only using @deprecation alone.
> > And
> > > > > there are additional benefits with @Retired.
> > > > >
> > > > > Just my 2 cents and looking forward to your feedback, especially
> > > > different
> > > > > opinions that will help me understand the issue better. Thanks!
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > <sr...@confluent.io.invalid> wrote:
> > > > >
> > > > > > Hi Xintong,
> > > > > >
> > > > > > Thanks for the summary, most of the points that you agreed upon
> > also
> > > > make
> > > > > > sense to me.
> > > > > >
> > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > demoting
> > > > > APIs
> > > > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > > > version
> > > > > > bump,
> > > > > > >   are not good practices.
> > > > > >
> > > > > > I hope we can can go beyond calling it “not a good practice" and
> > > reach
> > > > > > consensus that we will not demote or remove public APIs in minor
> > > > releases
> > > > > > and (technically) enforce this rule for all contributions. I
> think
> > > > it’s a
> > > > > > reasonable expectation for stability from a project as mature as
> > > Flink.
> > > > > >
> > > > > > > I'm personally in favor of option 1. As the migration-period
> rule
> > > is
> > > > > > newly
> > > > > > > proposed, I think it's fair to make exceptions for cases where
> we
> > > > > already
> > > > > > > missed the best chance for planning the deprecation. Moreover,
> I
> > do
> > > > > > believe
> > > > > > > having a quick major version bump does confuse users. Yes, we
> can
> > > > > explain
> > > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> > other
> > > > than
> > > > > > the
> > > > > > > removal of an deprecated API. But having to explain this itself
> > is
> > > an
> > > > > > > indicator that it might be confusing for users.
> > > > > > >
> > > > > > >
> > > > > > > Becket, on the other hand, prefers option 2. From my
> > understanding,
> > > > his
> > > > > > > major point is that a quick major version bump causes barely
> any
> > > > actual
> > > > > > > lose on users, while in option 1 not providing the migration
> > period
> > > > or
> > > > > > > providing a shorter on is an actual lose on users. (@Becket,
> > please
> > > > > > correct
> > > > > > > me if I'm mistaken.)
> > > > > > >
> > > > > >
> > > > > > I would be open to both options and maybe postpone this decision
> > > until
> > > > we
> > > > > > have the complete roadmap for Flink 2.0. We would have a better
> > > > overview
> > > > > > about the technical consequences, e.g. how hard it would be to
> > offer
> > > > and
> > > > > > maintain both APIs side by side, how well we get both APIs
> > separated,
> > > > and
> > > > > > whether or not we will be able to use the full potential of our
> > > > breaking
> > > > > > changes before the DataStream API is completely removed. One
> > example
> > > > for
> > > > > > the last point I can think of would be moving from lazy to eager
> > > state
> > > > > > declaration. If we believe that it is reasonable to support all
> > > planned
> > > > > > changes while maintaining both APIs, I'm leaning towards option
> 2.
> > > > > >
> > > > > > Best,
> > > > > > Stefan
> > > > > >
> > > > > >
> > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <tonysong820@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > Becket and I had an offline voice call earlier today. We have
> > > reached
> > > > > > > consensus on some of the arguments, but still hold different
> > > opinions
> > > > > on
> > > > > > > some others. I'd like to post the outcome here for
> transparency.
> > > > > > >
> > > > > > > We both agree that:
> > > > > > >
> > > > > > >   1. Providing a migration period would be beneficial for our
> > > users,
> > > > > and
> > > > > > >   we should do that
> > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > demoting
> > > > > APIs
> > > > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > > > version
> > > > > > bump,
> > > > > > >   are not good practices.
> > > > > > >   3. Ideally, with this FLIP, developers should be more careful
> > and
> > > > > plan
> > > > > > >   API changes ahead. That means:
> > > > > > >      1. Be more careful with designing APIs and promoting them
> to
> > > > > Public,
> > > > > > >      so that they won't be changed / removed very soon, and
> also
> > > the
> > > > > > >      maintenance overhead for keeping them after deprecation
> > should
> > > > be
> > > > > > >      affordable. (I believe this also aligns with John's
> > opinion.)
> > > > > > >      2. Plan the deprecation / removal of old APIs that we
> don't
> > > want
> > > > > to
> > > > > > >      carry to the next major version earlier, so that they'll
> > have
> > > > the
> > > > > > >      2-minor-release migration period before the next major
> > version
> > > > > bump.
> > > > > > >   4. A practical situation is that, we are not ready for
> > > deprecating
> > > > > the
> > > > > > >   DataStream API, nor afford to carry it for another couple of
> > > years.
> > > > > > >      1. We cannot deprecate it now, because the new
> > ProcessFunction
> > > > API
> > > > > > >      (the planned replacement for DataStream API) is not yet
> > ready.
> > > > We
> > > > > > haven't
> > > > > > >      planned the ProcessFunction API earlier because the
> original
> > > > > > > plan was to do
> > > > > > >      in-place changes directly on the DataStream API in release
> > > 2.0,
> > > > > > until the
> > > > > > >      smooth migration period is proposed. Before this, the only
> > > > > > guarantees we
> > > > > > >      provide for Public APIs is that they'll stay compatible
> > until
> > > > the
> > > > > > next
> > > > > > >      major version, which led to the understanding the in-place
> > > > changes
> > > > > > on
> > > > > > >      Public APIs should be fine.
> > > > > > >      2. We also cannot afford carrying the DataStream API for
> > > another
> > > > > > >      couple of years. One of the reasons that we want to
> > refactor /
> > > > > > replace
> > > > > > >      DataStream API is that it exposes and depends on too many
> > > > > > > Flink's internal
> > > > > > >      runtime implementations, which has already limited /
> blocked
> > > > plans
> > > > > > for
> > > > > > >      Flink's internal improvements for many times. Also due to
> > the
> > > > wide
> > > > > > and
> > > > > > >      transitive dependencies on the internals, maintaining two
> > sets
> > > > of
> > > > > > lower
> > > > > > >      APIs itself is expensive and would be better to keep it
> > short.
> > > > > This
> > > > > > is
> > > > > > >      admittedly a historical design flaw which should be
> avoided
> > in
> > > > > > future.
> > > > > > >
> > > > > > > What we haven't reached consensus is about how to deal with the
> > > > current
> > > > > > > situation around the DataStream API. There are two options.
> > > > > > >
> > > > > > >
> > > > > > >   1. We say this FLIP is enforced starting release 2.0. For
> > current
> > > > 1.x
> > > > > > >   APIs, we provide a migration period with best effort, while
> > > > allowing
> > > > > > >   exceptions for immediate removal in 2.0. That means we will
> > still
> > > > try
> > > > > > with
> > > > > > >   best effort to get the ProcessFuncion API ready and deprecate
> > the
> > > > > > >   DataStream API in 1.x, but will also be allowed to remove
> > > > DataStream
> > > > > > API in
> > > > > > >   2.0 if it's not deprecated 2 minor releases before the major
> > > > version
> > > > > > bump.
> > > > > > >   2. We strictly follow the process in this FLIP, and will
> > quickly
> > > > bump
> > > > > > >   the major version from 2.x to 3.0 once the migration period
> for
> > > > > > DataStream
> > > > > > >   API is reached.
> > > > > > >
> > > > > > > I'm personally in favor of option 1. As the migration-period
> rule
> > > is
> > > > > > newly
> > > > > > > proposed, I think it's fair to make exceptions for cases where
> we
> > > > > already
> > > > > > > missed the best chance for planning the deprecation. Moreover,
> I
> > do
> > > > > > believe
> > > > > > > having a quick major version bump does confuse users. Yes, we
> can
> > > > > explain
> > > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> > other
> > > > than
> > > > > > the
> > > > > > > removal of an deprecated API. But having to explain this itself
> > is
> > > an
> > > > > > > indicator that it might be confusing for users.
> > > > > > >
> > > > > > >
> > > > > > > Becket, on the other hand, prefers option 2. From my
> > understanding,
> > > > his
> > > > > > > major point is that a quick major version bump causes barely
> any
> > > > actual
> > > > > > > lose on users, while in option 1 not providing the migration
> > period
> > > > or
> > > > > > > providing a shorter on is an actual lose on users. (@Becket,
> > please
> > > > > > correct
> > > > > > > me if I'm mistaken.)
> > > > > > >
> > > > > > >
> > > > > > > And we'd like to hear more feedback from the community.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > vvcephei@apache.org
> > > > > > <ma...@apache.org>> wrote:
> > > > > > >
> > > > > > >> Hi Becket and Xintong,
> > > > > > >>
> > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > >>
> > > > > > >> Once an API is marked Public, we're committing to support it
> > until
> > > > > it's
> > > > > > >> deprecated, and once it's deprecated, to leave it in place for
> > at
> > > > > least
> > > > > > two
> > > > > > >> minor releases and then only remove it in the following major
> > > > release.
> > > > > > >>
> > > > > > >> As a user, I would be dismayed to discover that the project
> > > > > maintainers
> > > > > > >> demoted a Public API to PublicEvolving or Experimental in
> order
> > to
> > > > > > violate
> > > > > > >> the spirit of the deprecation period for Public APIs.
> Likewise,
> > > I'd
> > > > > > view a
> > > > > > >> rapid sequence of minor releases for the purpose of dropping
> > > > > deprecated
> > > > > > >> APIs in the next major release as another violation of the
> > spirit
> > > of
> > > > > our
> > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > >>
> > > > > > >> I'm glad we're gaming out these situations before we institute
> > > this
> > > > > > FLIP,
> > > > > > >> and I hope we all converge on a clear understanding of what
> > we're
> > > > > > >> guaranteeing. From where I'm sitting, the essence of this FLIP
> > is
> > > > > > >> specifically to prevent us from doing whatever we want and
> > instead
> > > > to
> > > > > > >> commit to providing a suitable notice period to users who rely
> > on
> > > > > Public
> > > > > > >> APIs.
> > > > > > >>
> > > > > > >> It honestly shouldn't matter what the maintenance overhead is
> > (or,
> > > > > > rather,
> > > > > > >> evaluating the maintenance overhead should be a consideration
> > > before
> > > > > we
> > > > > > >> promote an API to Public to begin with). My experience in
> other
> > > > > > projects is
> > > > > > >> that regretting a feature so much you want to drop it asap is
> > > > > extremely
> > > > > > >> rare, and that living with the pain for a little while until
> the
> > > > > > >> deprecation period expires will probably make us better
> > engineers.
> > > > > > >>
> > > > > > >> With that all said, one thing that is not a violation is to
> > > propose
> > > > a
> > > > > > new
> > > > > > >> major release in order to drop especially burdensome
> deprecated
> > > APIs
> > > > > > that
> > > > > > >> have already enjoyed their full deprecation period. Those
> really
> > > bad
> > > > > > "case
> > > > > > >> A" features will be very rare.
> > > > > > >>
> > > > > > >> Looking over the FLIP again, I see a reference to the release
> > > > > > >> documentation, but not to the build tooling support for the
> > > > Deprecated
> > > > > > >> annotation, so it might help to share this scenario: The way
> > this
> > > > will
> > > > > > all
> > > > > > >> come together in practice is that, by giving good stability
> > > > > guarantees,
> > > > > > we
> > > > > > >> will encourage our users to bump minor versions when they are
> > > > > released,
> > > > > > >> which means that they will see compiler warnings immediately
> if
> > > any
> > > > of
> > > > > > the
> > > > > > >> APIs they've used are deprecated. In fact, I'd recommend for
> > them
> > > to
> > > > > use
> > > > > > >> the `-Werror` javac flag to be sure that they don't miss those
> > > > > warnings.
> > > > > > >> Given our release cadence, this means that they will have
> about
> > > six
> > > > > > months
> > > > > > >> to migrate off of any deprecated APIs without being forced to
> > > avoid
> > > > > > >> updates. And they really need to be able to adopt further
> minor
> > > > > updates
> > > > > > in
> > > > > > >> the series, since they may contain further improvements that
> > > > actually
> > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > >>
> > > > > > >> The compiler warnings are a far more reliable mechanism to
> > > advertise
> > > > > > >> deprecation than release notes, since users' own builds will
> > > notify
> > > > > them
> > > > > > >> right away about only the deprecations that are relevant to
> > them,
> > > > but
> > > > > it
> > > > > > >> does rely on users feeling ok to bump up minor versions in a
> > > timely
> > > > > > >> fashion. If we consistently burn them by not observing the
> > > stability
> > > > > > >> guarantees, they'll try to avoid updates instead, and the
> whole
> > > > thing
> > > > > > falls
> > > > > > >> apart.
> > > > > > >>
> > > > > > >> Thanks again,
> > > > > > >> -John
> > > > > > >>
> > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > >>> Hi Xintong,
> > > > > > >>>
> > > > > > >>> Let's compare the following cases:
> > > > > > >>>
> > > > > > >>> A. If the maintenance overhead of the deprecated API is high,
> > and
> > > > we
> > > > > > want
> > > > > > >>> to remove it after two minor releases. Then there are two
> > > options:
> > > > > > >>>    A1: Demote the API in the major version bump and remove
> the
> > > code
> > > > > > >> with a
> > > > > > >>> minor version bump.
> > > > > > >>>    A2: Do exactly the same as 1-A, except that when removing
> > the
> > > > > code,
> > > > > > >>> bump up the major version. So this might be a short major
> > > version.
> > > > > > >>>
> > > > > > >>> B. If the maintenance overhead of the deprecated API is not
> > high,
> > > > > > >> therefore
> > > > > > >>> keeping it for a long time is affordable. There are also two
> > > > options:
> > > > > > >>>    B1: Same as A-1, demote the API in the major version bump
> > and
> > > > > remove
> > > > > > >>> the code with a minor version bump.
> > > > > > >>>    B2: Keep the API for all the minor versions in the major
> > > > version,
> > > > > > and
> > > > > > >>> only remove the code in the next major version.
> > > > > > >>>
> > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > >>>
> > > > > > >>> For case A:
> > > > > > >>> The following stays the same for A1 and A2:
> > > > > > >>>  - users will lose the API after two minor leases. So the
> > > migration
> > > > > > >> period
> > > > > > >>> is the same for A-1 and A-2.
> > > > > > >>>  - A1 and A2 will have exactly the same code after the
> removal
> > of
> > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > >>>  - To move forward, users need to move to the next minor
> > version
> > > in
> > > > > A1,
> > > > > > >> or
> > > > > > >>> to the next major version in A2. Because the code are the
> same,
> > > the
> > > > > > >> actual
> > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > >>>
> > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > >>>  - A1 allows keeping the major version release cadence. A-2
> > will
> > > > > have a
> > > > > > >>> short major version release.
> > > > > > >>>  - A1 breaks the well understood API semantic, while A-2 does
> > > not.
> > > > > > >>>
> > > > > > >>> From what I see, since there is no well establish standard
> > > > regarding
> > > > > > how
> > > > > > >>> long a major version should be released, A short major
> version
> > > > > release
> > > > > > is
> > > > > > >>> potential and emotional. It is not ideal but does not have
> > > material
> > > > > > >>> downsides compared with A1. I did not hear anyone complaining
> > > about
> > > > > > Kafka
> > > > > > >>> only has two 1.x release. However, A1 actually breaks the
> well
> > > > > > understood
> > > > > > >>> API semantic, which has more material impact.
> > > > > > >>>
> > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should fall
> > into
> > > > > case
> > > > > > B.
> > > > > > >>> So, short major versions should be very occasional. I'd be
> very
> > > > > > concerned
> > > > > > >>> if the reason we choose A1 is simply because we cannot afford
> > > > > > >> maintaining a
> > > > > > >>> bunch of deprecated APIs until the next major version. This
> > > > indicates
> > > > > > >> that
> > > > > > >>> the actual problem we need to solve is to lower the
> maintenance
> > > > > > overhead
> > > > > > >> of
> > > > > > >>> deprecated APIs, so that we are comfortable to keep them
> > longer.
> > > As
> > > > > > John
> > > > > > >>> and I mentioned earlier, there are ways to achieve this and
> we
> > > need
> > > > > to
> > > > > > >>> learn how to do it in Flink. Otherwise, our discussion about
> > > > > versioning
> > > > > > >>> here does not bring much value, because we will end up with a
> > > bunch
> > > > > of
> > > > > > >>> short-lived APIs which upset our users, no matter how we
> > version
> > > > the
> > > > > > >>> releases.
> > > > > > >>>
> > > > > > >>> So, if there are concrete examples that you think will block
> us
> > > > from
> > > > > > >>> keeping API stability with affordable cost, let's take a look
> > > > > together
> > > > > > >> and
> > > > > > >>> see if that can be improved.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>>
> > > > > > >>> Jiangjie (Becket) Qin
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> From what I see,
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > tonysong820@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>>>>
> > > > > > >>>>> The part I don't understand is if we are willing to have a
> > > > > migration
> > > > > > >>>>> period, and do a minor version bump to remove an API, what
> do
> > > we
> > > > > > >> lose to
> > > > > > >>>> do
> > > > > > >>>>> a major version bump instead, so we don't break the common
> > > > > versioning
> > > > > > >>>>> semantic?
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> I think we are talking about the cases where a major version
> > > bump
> > > > > > >> happens
> > > > > > >>>> before a deprecated Public API reaches its migration period.
> > So
> > > > > > >> removing
> > > > > > >>>> the API with another major version bump means we have two
> > > > > consecutive
> > > > > > >> major
> > > > > > >>>> versions in a very short time. And as previously mentioned,
> > > having
> > > > > > >> frequent
> > > > > > >>>> major version bumps would weaken the value of the commitment
> > > > "Public
> > > > > > >> API
> > > > > > >>>> stay compatible within a major version". I think users also
> > have
> > > > > > >>>> expectations about how long a major version should live,
> which
> > > > > should
> > > > > > >> be at
> > > > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > > > >>>>
> > > > > > >>>> This is another option, but I think it is very likely more
> > > > expensive
> > > > > > >> than
> > > > > > >>>>> simply bumping the major version. And I imagine there will
> be
> > > > > > >> questions
> > > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> > 2.0"?
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> Yes, it's more expensive than simply bumping the major
> > version,
> > > > but
> > > > > > >> IMHO
> > > > > > >>>> it's probably cheaper than carrying the API until the next
> > major
> > > > > > >> version if
> > > > > > >>>> we don't bump the major version very soon. And see my reply
> > > above
> > > > on
> > > > > > >> why
> > > > > > >>>> not bumping immediately.
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>>
> > > > > > >>>> Xintong
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > becket.qin@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > >> wrote:
> > > > > > >>>>
> > > > > > >>>>> Hi Xintong,
> > > > > > >>>>>
> > > > > > >>>>> Please see the replies below.
> > > > > > >>>>>
> > > > > > >>>>> I see your point, that users would feel surprised if they
> > find
> > > > > > >> things no
> > > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > > However,
> > > > > > >> I'd
> > > > > > >>>>> like
> > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> similar
> > > > > > >> problem
> > > > > > >>>>>> anyway. So the question is, how do we catch users'
> attention
> > > and
> > > > > > >> make
> > > > > > >>>>> sure
> > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer
> be
> > > > Public
> > > > > > >> in
> > > > > > >>>>> 2.0.
> > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > warnings
> > > > in
> > > > > > >>>> logs,
> > > > > > >>>>>> etc.
> > > > > > >>>>>
> > > > > > >>>>> First of all, I am not a fan of removing PublicEvolving
> APIs
> > in
> > > > > minor
> > > > > > >>>>> version changes. Personally speaking, I want them to be
> also
> > > > > removed
> > > > > > >> in
> > > > > > >>>>> major version changes.
> > > > > > >>>>>
> > > > > > >>>>> Empirically, I rarely see projects with complex API
> semantic
> > > work
> > > > > > >> well.
> > > > > > >>>>> Many, if not most, users don't read docs / release notes /
> > > > warnings
> > > > > > >> in
> > > > > > >>>>> logs. I don't think that is their fault. Typically an
> > > application
> > > > > > >>>> developer
> > > > > > >>>>> will have to deal with dozens of libraries maintained by
> > > various
> > > > > > >>>>> communities / groups. It is just too difficult for them to
> > keep
> > > > > > >> track of
> > > > > > >>>>> all the specific semantics each project puts there. If you
> > > think
> > > > > > >> about
> > > > > > >>>> it,
> > > > > > >>>>> how many of Flink developers read all the release notes of
> > > Guava,
> > > > > > >> Apache
> > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a
> > version
> > > > of
> > > > > > >> them?
> > > > > > >>>>> One would probably try bumping the dependency version and
> see
> > > if
> > > > > > >> there is
> > > > > > >>>>> an exception and solve them case by case. And if it is a
> > minor
> > > > > > >> version
> > > > > > >>>>> bump, one probably does not expect an exception at all.
> > Another
> > > > > > >> example
> > > > > > >>>> is
> > > > > > >>>>> that in Flink we still have so many usage of deprecated
> > methods
> > > > all
> > > > > > >> over
> > > > > > >>>>> the place, and I strongly doubt everyone knows when the
> > source
> > > > code
> > > > > > >> of
> > > > > > >>>>> these methods are deprecated and when they should be
> removed.
> > > > > > >>>>>
> > > > > > >>>>> So, the most important principle of API is simple and
> > > intuitive.
> > > > > The
> > > > > > >>>>> versioning semantic is a simple and universally accepted
> API
> > > > > > >> stability
> > > > > > >>>>> standard. If we ourselves as Flink developers are relying
> on
> > > this
> > > > > > >> for our
> > > > > > >>>>> own dependencies. I don't think we can expect more from our
> > > > users.
> > > > > > >>>>>
> > > > > > >>>>> Another possible alternative: whenever there's a deprecated
> > > > Public
> > > > > > >> API
> > > > > > >>>> that
> > > > > > >>>>>> reaches a major version bump before the migration period,
> > and
> > > we
> > > > > > >> also
> > > > > > >>>>> don't
> > > > > > >>>>>> want to carry it for all the next major release series, we
> > may
> > > > > > >> consider
> > > > > > >>>>>> releasing more minor releases for the previous major
> version
> > > > after
> > > > > > >> the
> > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then
> we
> > > > bump
> > > > > > >> to
> > > > > > >>>> 2.0,
> > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> provide
> > > > users
> > > > > > >>>> another
> > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > > > >>>> 2-minor-release
> > > > > > >>>>>> migration period.
> > > > > > >>>>>
> > > > > > >>>>> This is another option, but I think it is very likely more
> > > > > expensive
> > > > > > >> than
> > > > > > >>>>> simply bumping the major version. And I imagine there will
> be
> > > > > > >> questions
> > > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> > 2.0"?
> > > > > > >>>>>
> > > > > > >>>>> I think my major point is, we should not carry APIs
> > deprecated
> > > > in a
> > > > > > >>>>>> previous major version along all the next major version
> > > series.
> > > > > I'd
> > > > > > >>>> like
> > > > > > >>>>> to
> > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > period,
> > > as
> > > > > > >> long
> > > > > > >>>> as
> > > > > > >>>>> it
> > > > > > >>>>>> does not prevent us from making breaking changes. If it
> > > doesn't
> > > > > > >> work,
> > > > > > >>>> I'd
> > > > > > >>>>>> be in favor of not providing the migration period, but
> > > fallback
> > > > to
> > > > > > >> only
> > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > >>>>>
> > > > > > >>>>> The part I don't understand is if we are willing to have a
> > > > > migration
> > > > > > >>>>> period, and do a minor version bump to remove an API, what
> do
> > > we
> > > > > > >> lose to
> > > > > > >>>> do
> > > > > > >>>>> a major version bump instead, so we don't break the common
> > > > > versioning
> > > > > > >>>>> semantic?
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>>
> > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > tonysong820@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> As an end user who only uses Public APIs, if I don't
> change
> > > my
> > > > > > >> code
> > > > > > >>>> at
> > > > > > >>>>>>> all, my expectation is the following:
> > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I
> am
> > > fine
> > > > > > >> with
> > > > > > >>>>> all
> > > > > > >>>>>>> the 2.x versions.
> > > > > > >>>>>>> Actually I think there are some dependency version
> > resolution
> > > > > > >>>> policies
> > > > > > >>>>>> out
> > > > > > >>>>>>> there which picks the highest minor version when the
> > > > dependencies
> > > > > > >>>> pull
> > > > > > >>>>> in
> > > > > > >>>>>>> multiple minor versions of the same jar, which may be
> > broken
> > > if
> > > > > > >> we
> > > > > > >>>>> remove
> > > > > > >>>>>>> the API in minor releases.
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> I see your point, that users would feel surprised if they
> > find
> > > > > > >> things
> > > > > > >>>> no
> > > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > > However,
> > > > > > >> I'd
> > > > > > >>>>> like
> > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> similar
> > > > > > >> problem
> > > > > > >>>>>> anyway. So the question is, how do we catch users'
> attention
> > > and
> > > > > > >> make
> > > > > > >>>>> sure
> > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer
> be
> > > > Public
> > > > > > >> in
> > > > > > >>>>> 2.0.
> > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > warnings
> > > > in
> > > > > > >>>> logs,
> > > > > > >>>>>> etc.
> > > > > > >>>>>>
> > > > > > >>>>>> Another possible alternative: whenever there's a
> deprecated
> > > > Public
> > > > > > >> API
> > > > > > >>>>> that
> > > > > > >>>>>> reaches a major version bump before the migration period,
> > and
> > > we
> > > > > > >> also
> > > > > > >>>>> don't
> > > > > > >>>>>> want to carry it for all the next major release series, we
> > may
> > > > > > >> consider
> > > > > > >>>>>> releasing more minor releases for the previous major
> version
> > > > after
> > > > > > >> the
> > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then
> we
> > > > bump
> > > > > > >> to
> > > > > > >>>> 2.0,
> > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> provide
> > > > users
> > > > > > >>>> another
> > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > > > >>>> 2-minor-release
> > > > > > >>>>>> migration period.
> > > > > > >>>>>>
> > > > > > >>>>>> I think my major point is, we should not carry APIs
> > deprecated
> > > > in
> > > > > a
> > > > > > >>>>>> previous major version along all the next major version
> > > series.
> > > > > I'd
> > > > > > >>>> like
> > > > > > >>>>> to
> > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > period,
> > > as
> > > > > > >> long
> > > > > > >>>> as
> > > > > > >>>>> it
> > > > > > >>>>>> does not prevent us from making breaking changes. If it
> > > doesn't
> > > > > > >> work,
> > > > > > >>>> I'd
> > > > > > >>>>>> be in favor of not providing the migration period, but
> > > fallback
> > > > to
> > > > > > >> only
> > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > >>>>>>
> > > > > > >>>>>> Best,
> > > > > > >>>>>>
> > > > > > >>>>>> Xintong
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > vvcephei@apache.org
> > > > > > <ma...@apache.org>
> > > > > > >>>
> > > > > > >>>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi Becket,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> conversation
> > > > about
> > > > > > >>>>>>> compatibility outside of this FLIP thread, but for now, I
> > can
> > > > > > >> accept
> > > > > > >>>>> your
> > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks again,
> > > > > > >>>>>>> John
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > >>>>>>>> Hi John,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in major
> > > > releases
> > > > > > >>>>> across
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> board? I understand that Experimental and
> PublicEvolving
> > > APIs
> > > > > > >> are
> > > > > > >>>> by
> > > > > > >>>>>>>>> definition less stable, but it seems like this should
> be
> > > > > > >> reflected
> > > > > > >>>>> in
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must
> keep
> > > > them
> > > > > > >>>>> around
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> drop
> > > them
> > > > > > >> in a
> > > > > > >>>>>>> minor or
> > > > > > >>>>>>>>> patch release.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Personally speaking, I would love to do this, for
> exactly
> > > the
> > > > > > >>>> reason
> > > > > > >>>>>> you
> > > > > > >>>>>>>> mentioned. However, I did not propose this due to the
> > > > following
> > > > > > >>>>>> reasons:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> > accepted
> > > > > > >> FLIPs
> > > > > > >>>> too
> > > > > > >>>>>>> soon.
> > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > development.
> > > At
> > > > > > >> this
> > > > > > >>>>>>> point,
> > > > > > >>>>>>>> Flink still lacks some design / routines to support good
> > API
> > > > > > >>>>>>> evolvability /
> > > > > > >>>>>>>> extensibility. Just like you said, it takes some time to
> > be
> > > > > > >> good at
> > > > > > >>>>>> this.
> > > > > > >>>>>>>> In this case, my concern is that only removing
> > Experimental
> > > /
> > > > > > >>>>>>>> PublicEvolving APIs in major version changes may result
> in
> > > too
> > > > > > >> much
> > > > > > >>>>>>>> overhead and dramatically slow down the development of
> > > Flink.
> > > > > > >> So, I
> > > > > > >>>>> was
> > > > > > >>>>>>>> thinking that we can start with the current status.
> > > Hopefully
> > > > > > >> after
> > > > > > >>>>> we
> > > > > > >>>>>>> are
> > > > > > >>>>>>>> more comfortable with the maintenance overhead of
> > deprecated
> > > > > > >> APIs,
> > > > > > >>>> we
> > > > > > >>>>>> can
> > > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > > >> PublicEvolving
> > > > > > >>>>> APIs.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Hi Becket,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is
> > > really
> > > > > > >>>>>> important.
> > > > > > >>>>>>> I
> > > > > > >>>>>>>>> understand some people’s concerns about the additional
> > > burden
> > > > > > >> for
> > > > > > >>>>>>> project
> > > > > > >>>>>>>>> maintainers, but my personal experience with Kafka has
> > been
> > > > > > >> that
> > > > > > >>>>> it’s
> > > > > > >>>>>>> very
> > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to users.
> > In
> > > > > > >> fact,
> > > > > > >>>>> users
> > > > > > >>>>>>>>> being able to confidently upgrade is also a benefit to
> > > > > > >>>> maintainers,
> > > > > > >>>>> as
> > > > > > >>>>>>> we
> > > > > > >>>>>>>>> will get fewer questions from people stuck on very old
> > > > > > >> versions.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> One question:
> > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in major
> > > > > > >> releases
> > > > > > >>>>> across
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> board? I understand that Experimental and
> PublicEvolving
> > > APIs
> > > > > > >> are
> > > > > > >>>> by
> > > > > > >>>>>>>>> definition less stable, but it seems like this should
> be
> > > > > > >> reflected
> > > > > > >>>>> in
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must
> keep
> > > > them
> > > > > > >>>>> around
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> drop
> > > them
> > > > > > >> in a
> > > > > > >>>>>>> minor or
> > > > > > >>>>>>>>> patch release.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The advantage of forbidding the removal of any API in
> > minor
> > > > or
> > > > > > >>>> patch
> > > > > > >>>>>>>>> releases is that users will get a strong guarantee that
> > > they
> > > > > > >> can
> > > > > > >>>>> bump
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> minor or patch version and still be able to compile, or
> > > even
> > > > > > >> just
> > > > > > >>>>>>> re-link
> > > > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at
> > run
> > > > > > >> time.
> > > > > > >>>>> This
> > > > > > >>>>>>> is a
> > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> Experimental
> > > > APIs
> > > > > > >>>>> outside
> > > > > > >>>>>>> of
> > > > > > >>>>>>>>> major releases, users can no longer confidently
> upgrade.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of
> > > points:
> > > > > > >>>>>>>>> * I’d use the official Deprecated annotation instead of
> > > > > > >>>> introducing
> > > > > > >>>>>> our
> > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > > > integrated
> > > > > > >>>> into
> > > > > > >>>>>>> build
> > > > > > >>>>>>>>> tools and IDEs.
> > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this
> FLIP;
> > > it
> > > > > > >> seems
> > > > > > >>>>>>>>> orthogonal, and something that should probably be taken
> > > > > > >>>> case-by-case
> > > > > > >>>>>>>>> anyway.
> > > > > > >>>>>>>>> * Aside from deprecation and removal, there have been
> > some
> > > > > > >>>>> discussions
> > > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible
> ways.
> > > > > > >> This is
> > > > > > >>>>>>> somewhat
> > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it
> before,
> > > > it’ll
> > > > > > >>>> take
> > > > > > >>>>>> some
> > > > > > >>>>>>>>> time to become good at it. I feel like this topic
> should
> > > also
> > > > > > >> be
> > > > > > >>>>>>> orthogonal
> > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to
> adopt a
> > > > > > >> simple
> > > > > > >>>>>> policy
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>> to break existing user programs, and leave the “how” up
> > to
> > > > > > >>>>>> implementers
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>> reviewers.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks again,
> > > > > > >>>>>>>>> John
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > >>>>>>>>>> Hi All,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is
> a
> > > > > > >> great
> > > > > > >>>>> idea.
> > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > > > >>>> @PublicEvolving
> > > > > > >>>>>> --(2
> > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > > > > > >> flexibility
> > > > > > >>>>>>> without
> > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> > > > > > >> change
> > > > > > >>>>>> between
> > > > > > >>>>>>>>> major
> > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> > provides
> > > > > > >>>>> additional
> > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> > themself
> > > > > > >> are
> > > > > > >>>> not
> > > > > > >>>>>>>>> changed.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> issues. I
> > > > > > >> want to
> > > > > > >>>>>> move
> > > > > > >>>>>>> one
> > > > > > >>>>>>>>>> step further and suggest introducing one more
> annotation
> > > > > > >>>> @Retired.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > > > > > >> downgrading
> > > > > > >>>>>>> @Public
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply,
> > Java
> > > > > > >>>> standard
> > > > > > >>>>>>>>>> @deprecated should be used in the early stage of the
> > > > > > >> deprecation
> > > > > > >>>>>>> process
> > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java
> does
> > > not
> > > > > > >>>> allow
> > > > > > >>>>>> us
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>> extend annotation, I think it would be feasible to
> have
> > > the
> > > > > > >> new
> > > > > > >>>>>>> @Retired
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>> help us monitor and manage the deprecation process,
> > house
> > > > > > >>>>> cleaning,
> > > > > > >>>>>>> etc.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> @Retired:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > > > >> compatibility
> > > > > > >>>>>> before
> > > > > > >>>>>>>>> APIs
> > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> > processes
> > > > > > >> on
> > > > > > >>>> the
> > > > > > >>>>> ML
> > > > > > >>>>>>> are
> > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1
> minor
> > > > > > >> release
> > > > > > >>>>>>> sharply
> > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check
> whether
> > > it
> > > > > > >> is
> > > > > > >>>>>>> possible).
> > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to
> > remove
> > > > > > >> them.
> > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> > > @Retired
> > > > > > >> are
> > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal
> ->
> > > > > > >> @Retired
> > > > > > >>>>>> could
> > > > > > >>>>>>>>> also
> > > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > > >>>> dev-friendliness,
> > > > > > >>>>>> but
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>>> mandatory.
> > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to
> support
> > > the
> > > > > > >>>>>>> deprecation
> > > > > > >>>>>>>>>> process management. Further extension is possible,
> since
> > > the
> > > > > > >>>>>>> annotation
> > > > > > >>>>>>>>> is
> > > > > > >>>>>>>>>> built by us.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best regards,
> > > > > > >>>>>>>>>> Jing
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > > >>>>>>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies
> > inline
> > > > > > >>>> below.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public
> > API
> > > > > > >> is
> > > > > > >>>>> also a
> > > > > > >>>>>>>>> kind of
> > > > > > >>>>>>>>>>>> such change, just like removing one, which can only
> > > > > > >> happen
> > > > > > >>>> with
> > > > > > >>>>>>> major
> > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > > Public
> > > > > > >>>> APIs
> > > > > > >>>>>>>>> anytime,
> > > > > > >>>>>>>>>>> but
> > > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > > >> reaching
> > > > > > >>>> the
> > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > > > >> would
> > > > > > >>>> be a
> > > > > > >>>>>>> weaker
> > > > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > > > >> major
> > > > > > >>>>>> version
> > > > > > >>>>>>>>> bumps,
> > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > 2-minor-release
> > > > > > >>>>>> migration
> > > > > > >>>>>>>>>>> period.
> > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> `@PublicEvolving`
> > > > > > >> sounds
> > > > > > >>>>>>> against
> > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > > >> necessary,
> > > > > > >>>>>> e.g.,
> > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> > > > > > >> change my
> > > > > > >>>>> code
> > > > > > >>>>>>> at
> > > > > > >>>>>>>>> all,
> > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> issue, I
> > > am
> > > > > > >> fine
> > > > > > >>>>>> with
> > > > > > >>>>>>> all
> > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > >>>>>>>>>>> Actually I think there are some dependency version
> > > > > > >> resolution
> > > > > > >>>>>>> policies
> > > > > > >>>>>>>>> out
> > > > > > >>>>>>>>>>> there which picks the highest minor version when the
> > > > > > >>>> dependencies
> > > > > > >>>>>>> pull
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which may be
> > > > > > >> broken if
> > > > > > >>>>> we
> > > > > > >>>>>>>>> remove
> > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > > that
> > > > > > >> we
> > > > > > >>>>> bump
> > > > > > >>>>>> up
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > > > >> However,
> > > > > > >>>>> this
> > > > > > >>>>>>> also
> > > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> > > will
> > > > > > >>>> stay
> > > > > > >>>>>>> stable
> > > > > > >>>>>>>>>>>> within the major release series, as the series can
> end
> > > > > > >>>> anytime.
> > > > > > >>>>>>> IMHO,
> > > > > > >>>>>>>>>>> short
> > > > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > > > >> users
> > > > > > >>>>>> happy",
> > > > > > >>>>>>> but
> > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > > >> frequent
> > > > > > >>>>>> breaking
> > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> approach,
> > we
> > > > > > >> can
> > > > > > >>>>>> still
> > > > > > >>>>>>>>> have
> > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > Public
> > > > > > >> APIs
> > > > > > >>>>>>>>> deprecated
> > > > > > >>>>>>>>>>> at
> > > > > > >>>>>>>>>>>> the end of the previous major version to be removed
> in
> > > > > > >> the
> > > > > > >>>> next
> > > > > > >>>>>>> major
> > > > > > >>>>>>>>>>>> version.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I totally agree that frequent major version bumps are
> > not
> > > > > > >>>> ideal,
> > > > > > >>>>>> but
> > > > > > >>>>>>>>> here
> > > > > > >>>>>>>>>>> we are comparing it with a minor version bump which
> > > > > > >> removes a
> > > > > > >>>>>> Public
> > > > > > >>>>>>>>> API.
> > > > > > >>>>>>>>>>> So the context is that we have already decided to
> > remove
> > > > > > >> this
> > > > > > >>>>>> Public
> > > > > > >>>>>>> API
> > > > > > >>>>>>>>>>> while keeping everything else backwards compatible. I
> > > > > > >> think a
> > > > > > >>>>> major
> > > > > > >>>>>>>>> version
> > > > > > >>>>>>>>>>> bump is a commonly understood signal here, compared
> > with
> > > a
> > > > > > >>>> minor
> > > > > > >>>>>>> version
> > > > > > >>>>>>>>>>> change. From end users' perspective, for those who
> are
> > > not
> > > > > > >>>>>> impacted,
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>>>> this case upgrading a major version is not
> necessarily
> > > more
> > > > > > >>>>>> involved
> > > > > > >>>>>>>>> than
> > > > > > >>>>>>>>>>> upgrading a minor version - both should be as smooth
> > as a
> > > > > > >>>>>> dependency
> > > > > > >>>>>>>>>>> version change. For those who are impacted, they will
> > > lose
> > > > > > >> the
> > > > > > >>>>>> Public
> > > > > > >>>>>>>>> API
> > > > > > >>>>>>>>>>> anyways and a major version bump ensures there is no
> > > > > > >> surprise.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Public API is a well defined common concept, and one
> > of
> > > > > > >> its
> > > > > > >>>>>>>>>>>>> convention is that it only changes with a major
> > version
> > > > > > >>>>> change.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a
> Public
> > > > > > >> API is
> > > > > > >>>>>> also a
> > > > > > >>>>>>>>> kind
> > > > > > >>>>>>>>>>>> of such change, just like removing one, which can
> only
> > > > > > >> happen
> > > > > > >>>>>> with
> > > > > > >>>>>>>>> major
> > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > > Public
> > > > > > >>>> APIs
> > > > > > >>>>>>>>> anytime,
> > > > > > >>>>>>>>>>> but
> > > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > > >> reaching
> > > > > > >>>> the
> > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > > > >> would
> > > > > > >>>> be a
> > > > > > >>>>>>> weaker
> > > > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > > > >> major
> > > > > > >>>>>> version
> > > > > > >>>>>>>>> bumps,
> > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > 2-minor-release
> > > > > > >>>>>> migration
> > > > > > >>>>>>>>>>> period.
> > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> `@PublicEvolving`
> > > > > > >> sounds
> > > > > > >>>>>>> against
> > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > > >> necessary,
> > > > > > >>>>>> e.g.,
> > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> But it should be
> > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if we
> > really
> > > > > > >> want
> > > > > > >>>>> to
> > > > > > >>>>>>> get
> > > > > > >>>>>>>>> rid
> > > > > > >>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely
> "legal"
> > > > > > >> that we
> > > > > > >>>>>> bump
> > > > > > >>>>>>> up
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > > > >> However,
> > > > > > >>>>> this
> > > > > > >>>>>>> also
> > > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> > > will
> > > > > > >>>> stay
> > > > > > >>>>>>> stable
> > > > > > >>>>>>>>>>>> within the major release series, as the series can
> end
> > > > > > >>>> anytime.
> > > > > > >>>>>>> IMHO,
> > > > > > >>>>>>>>>>> short
> > > > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > > > >> users
> > > > > > >>>>>> happy",
> > > > > > >>>>>>> but
> > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > > >> frequent
> > > > > > >>>>>> breaking
> > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> approach,
> > we
> > > > > > >> can
> > > > > > >>>>>> still
> > > > > > >>>>>>>>> have
> > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > Public
> > > > > > >> APIs
> > > > > > >>>>>>>>> deprecated
> > > > > > >>>>>>>>>>> at
> > > > > > >>>>>>>>>>>> the end of the previous major version to be removed
> in
> > > > > > >> the
> > > > > > >>>> next
> > > > > > >>>>>>> major
> > > > > > >>>>>>>>>>>> version.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Given our track record I would prefer a regular
> cycle
> > > > > > >> (1-2
> > > > > > >>>>> years)
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not
> put
> > > it
> > > > > > >>>> again
> > > > > > >>>>>> to
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> expectation
> > > on
> > > > > > >>>> when
> > > > > > >>>>>>>>> breaking
> > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good
> time
> > > > > > >> for new
> > > > > > >>>>>> major
> > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > > > > > >> needed.
> > > > > > >>>> That
> > > > > > >>>>>>> makes
> > > > > > >>>>>>>>> 1-2
> > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while
> > leaving
> > > > > > >> us
> > > > > > >>>> more
> > > > > > >>>>>>> time
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Xintong
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing
> minor
> > > > > > >>>>> versions
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>>>>>> previous
> > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> > > > > > >> viable
> > > > > > >>>> that
> > > > > > >>>>>> is
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>>>>>> practice.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > > > > > >> support
> > > > > > >>>> for
> > > > > > >>>>>> Java
> > > > > > >>>>>>>>> 8/11,
> > > > > > >>>>>>>>>>>>> which may very well be a problem for our current
> > users.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> confusion
> > > > > > >> about
> > > > > > >>>>>> Public
> > > > > > >>>>>>>>> API
> > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a
> mistake
> > > > > > >> in
> > > > > > >>>> the
> > > > > > >>>>>>> NOTES
> > > > > > >>>>>>>>> part
> > > > > > >>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just
> fixed
> > > > > > >> it.
> > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that
> we
> > > > > > >> are on
> > > > > > >>>>> the
> > > > > > >>>>>>> same
> > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > > > > > >> really
> > > > > > >>>> want
> > > > > > >>>>>> to
> > > > > > >>>>>>> get
> > > > > > >>>>>>>>>>> rid
> > > > > > >>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to
> get
> > us
> > > > > > >> to
> > > > > > >>>>> 2.0.
> > > > > > >>>>>> ;)
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the
> > cadence
> > > > > > >> of
> > > > > > >>>>> major
> > > > > > >>>>>>>>> releases
> > > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> > opinions
> > > > > > >> on
> > > > > > >>>>> that.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases
> > > between
> > > > > > >>>> major
> > > > > > >>>>>>> ones,
> > > > > > >>>>>>>>>>>>> that for us means that users have to potentially
> deal
> > > > > > >> with
> > > > > > >>>>>>> breaking
> > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular
> cycle
> > > > > > >> (1-2
> > > > > > >>>>>> years)
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not
> put
> > > it
> > > > > > >>>> again
> > > > > > >>>>>> to
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> expectation
> > > on
> > > > > > >>>> when
> > > > > > >>>>>>>>> breaking
> > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate
> thread.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in
> mind?
> > > > > > >> Not
> > > > > > >>>>> sure
> > > > > > >>>>>> I
> > > > > > >>>>>>>>> fully
> > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change here.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> performance
> > in
> > > > > > >>>>> certain
> > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> unknowingly)
> > > > > > >> relied
> > > > > > >>>>> upon
> > > > > > >>>>>>>>>>>>> (
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > ),
> > > > > > a semantic change to some
> > > > > > >> API.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> submission.
> > A
> > > > > > >> few
> > > > > > >>>>>>> releases
> > > > > > >>>>>>>>> back
> > > > > > >>>>>>>>>>>>> we made changes such that the initialization of the
> > job
> > > > > > >>>> master
> > > > > > >>>>>>>>> happens
> > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > >>>>>>>>>>>>> This meant the job submission call returns sooner,
> > and
> > > > > > >> the
> > > > > > >>>> job
> > > > > > >>>>>>> state
> > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but
> > the
> > > > > > >>>>> observed
> > > > > > >>>>>>>>> behavior
> > > > > > >>>>>>>>>>>>> may be different.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over time we
> > > > > > >> changed
> > > > > > >>>>> what
> > > > > > >>>>>>> some
> > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jark Wu <im...@gmail.com>.
I agree with Jingsong and Becket.

Look at the legacy SourceFunction (a small part of DataStream API),
the SourceFunction is still not and can't be marked deprecated[1] until
now after the new Source was released 2 years ago, because the new Source
still can't fully consume the abilities of legacy API. Considering
DataStream
API is the most fundamental and complex API of Flink, I think it is worth
a longer time than the general process for the deprecation period to
wait for the new API be mature. The above 2 options sound a bit of rush
for such a widely used API.

I fully understand the concern of maintenance overhead, but it's a bit hard
for others to estimate maintenance costs without a concrete design and code
of the new ProcessFunction API. I agree with Becket that maybe we can
re-evaluate the API deprecation process once we have the new ProcessFunction
API. If the maintenance is indeed huge, I think it is reasonable to have a
special rule for this case at that time.

Best,
Jark


[1]: https://issues.apache.org/jira/browse/FLINK-28045

On Sun, 25 Jun 2023 at 16:22, Becket Qin <be...@gmail.com> wrote:

> Hi Jingsong,
>
> Thanks for the reply. I completely agree with you.
>
> The above 2 options are based on the assumption that the community cannot
> afford to maintain the deprecated DataStream API for long. I'd say we
> should try everything we can to maintain it for as much time as possible.
> DataStream API is actually the most used API in Flink by so many users at
> this point. Removing it any time soon will dramatically hurt our users. So
> ideally we should keep it for at least 2 years after deprecation, if not
> more.
>
> The prohibitively high maintenance overhead is just an assumption.
> Personally speaking, I don't feel this assumption is necessarily true. We
> should re-evaluate once we have the new ProcessFunction API in place.
> Without the code it is hard to tell for sure. I am actually kind of
> optimistic about the maintenance cost.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <ji...@gmail.com>
> wrote:
>
> > Thanks Becket and all for your discussion.
> >
> > > 1. We say this FLIP is enforced starting release 2.0. For current 1.x
> > APIs,
> > we provide a migration period with best effort, while allowing exceptions
> > for immediate removal in 2.0. That means we will still try with best
> effort
> > to get the ProcessFuncion API ready and deprecate the DataStream API in
> > 1.x, but will also be allowed to remove DataStream API in 2.0 if it's not
> > deprecated 2 minor releases before the major version bump.
> >
> > > 2. We strictly follow the process in this FLIP, and will quickly bump
> the
> > major version from 2.x to 3.0 once the migration period for DataStream
> API
> > is reached.
> >
> > Sorry, I didn't read the previous detailed discussion because the
> > discussion list was so long.
> >
> > I don't really like either of these options.
> >
> > Considering that DataStream is such an important API, can we offer a
> third
> > option:
> >
> > 3. Maintain the DataStream API throughout 2.X and remove it until 3.x.
> But
> > there's no need to assume that 2.X is a short version, it's still a
> normal
> > major version.
> >
> > Best,
> > Jingsong
> >
> > Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
> >
> > > Thanks much for the input, John, Stefan and Jing.
> > >
> > > I think Xingtong has well summarized the pros and cons of the two
> > options.
> > > Let's collect a few more opinions here and we can move forward with the
> > one
> > > more people prefer.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks Xingtong for the summary. If I could only choose one of the
> > given
> > > > two options, I would go with option 1. I understood that option 2
> > worked
> > > > great with Kafka. But the bridge release will still confuse users and
> > my
> > > > gut feeling is that many users will skip 2.0 and be waiting for 3.0
> or
> > > even
> > > > 3.x. And since fewer users will use Flink 2.x, the development focus
> > will
> > > > be on Flink 3.0 with the fact that the current Flink release is 1.17
> > and
> > > we
> > > > are preparing 2.0 release. That is weird for me.
> > > >
> > > > THB, I would not name the change from @Public to @Retired as a
> > demotion.
> > > > The purpose of @Retire is to extend the API lifecycle with one more
> > > stage,
> > > > like in the real world, people born, studied, graduated, worked, and
> > > > retired. Afaiu from the previous discussion, there are two rules we'd
> > > like
> > > > to follow simultaneously:
> > > >
> > > > 1. Public APIs can only be changed between major releases.
> > > > 2. A smooth migration phase should be offered to users, i.e. at
> least 2
> > > > minor releases after APIs are marked as @deprecated. There should be
> > new
> > > > APIs as the replacement.
> > > >
> > > > Agree, those rules are good to improve the user friendliness. Issues
> we
> > > > discussed are rising because we want to fulfill both of them. If we
> > take
> > > > care of deprecation very seriously, APIs can be marked as
> @Deprecated,
> > > only
> > > > when the new APIs as the replacement provide all functionalities the
> > > > deprecated APIs have. In an ideal case without critical bugs that
> might
> > > > stop users adopting the new APIs. Otherwise the expected
> "replacement"
> > > will
> > > > not happen. Users will still stick to the deprecated APIs, because
> the
> > > new
> > > > APIs can not be used. For big features, it will need at least 4 minor
> > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > >
> > > > - 1st minor release to build the new APIs as the replacement and
> > waiting
> > > > for feedback. It might be difficult to mark the old API as deprecated
> > in
> > > > this release, because we are not sure if the new APIs could cover
> 100%
> > > > functionalities.
> > > > -  In the lucky case,  mark all old APIs as deprecated in the 2nd
> minor
> > > > release. (I would even suggest having the new APIs released at least
> > for
> > > > two minor releases before marking it as deprecated to make sure they
> > can
> > > > really replace the old APIs, in case we care more about smooth
> > migration)
> > > > - 3rd minor release for the migration period
> > > > -  In another lucky case, the 4th release is a major release, the
> > > > deprecated APIs could be removed.
> > > >
> > > > The above described scenario works only in an ideal case. In reality,
> > it
> > > > might take longer to get the new APIs ready and mark the old API
> > > > deprecated. Furthermore, if the 4th release is not a major release,
> we
> > > will
> > > > have to maintain both APIs for many further minor releases. The
> > question
> > > is
> > > > how to know the next major release in advance, especially 4 minor
> > > releases'
> > > > period, i.e. more than 2 years in advance? Given that Flink contains
> > many
> > > > modules, it is difficult to ask devs to create a 2-3 years
> deprecation
> > > plan
> > > > for each case. In case we want to build major releases at a fast
> pace,
> > > > let's say every two years, it means devs must plan any API
> deprecation
> > > > right after each major release. Afaiac, it is quite difficult.
> > > >
> > > > The major issue is, afaiu, if we follow rule 2, we have to keep all
> > > @Public
> > > > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0.
> > > Then
> > > > we have to follow rule 1 to keep it unchanged until we have 3.0. That
> > is
> > > > why @Retired is useful to give devs more flexibility and still
> fulfill
> > > both
> > > > rules. Let's check it with examples:
> > > >
> > > > - we have @Public DataStream API in 1.18. It will not be marked
> > > > as @Deprecated, because the new APIs as the replacement are not
> ready.
> > > > - we keep the DataStream API itself unchanged in 2.0, but change the
> > > > annotation from @Public to @Retire. New APIs will be introduced too.
> In
> > > > this case, Rule 1 is ok, since public API is allowed to change
> between
> > > > major releases. Only changing annotation is the minimal change we
> could
> > > do
> > > > and it does not break rule 1. Rule 2 is ok too, since the DataStream
> > APIs
> > > > work exactly the same.  Attention: the change of @Public -> @Retired
> > can
> > > > only be done between major releases, because @Public APIs can only be
> > > > changed between major releases.
> > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > - in 2.2, DataStream will be kept for the migration period.
> > > > - in 2.3, DataStream will be removed.
> > > >
> > > > Becket mentioned previously (please correct me if I didn't understand
> > it
> > > > correctly) that users might not check the changes of annotation and
> the
> > > > upgrade process might not be smooth. I was wondering, if users don't
> > pay
> > > > attention to that, they will also not pay attention to @deprecated.
> The
> > > > migration will not be smooth too even when we stick to the Java
> > > > standard @deprecated. Let's go throw it with examples:
> > > >
> > > > Scenario 1: with @Retired
> > > >
> > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be
> > done
> > > > for users who still want to use DataStream. The migration is smooth.
> > But
> > > > they should be aware of the annotation changes, i.e. @Public ->
> > @Retired
> > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since
> > it
> > > is
> > > > a major release upgrade, breaking changes are expected.
> > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed. Since
> > > APIs
> > > > are marked as @Retired, breaking changes are expected too.
> > > >
> > > > In addition to fulfilling both rules, there are two more benefits of
> > this
> > > > option:
> > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > 2. no bridge release is required. DataStream API could be deprecated
> in
> > > 2.x
> > > > without breaking any of the two rules. @Retired provides enough
> > > flexibility
> > > > for future deprecations and upgrades.
> > > >
> > > > The only issue I can find is that users might be surprised in case 3,
> > if
> > > > they ignore the previous @Public -> @Retired change. But for such
> > users,
> > > > they will ignore @deprecated annotation too and will have issues with
> > > > pure @deprecated
> > > > annotation too:
> > > >
> > > > Scenario 2: with @deprecated
> > > >
> > > > The example will be slightly different from scenario 1, because
> @Public
> > > > APIs can only be removed between major releases. As an example,
> > > DataStream
> > > > API will be marked as deprecated in 1.19, kept in 1.20, and then
> > removed
> > > in
> > > > 2.0
> > > >
> > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as
> > > > deprecated are removed between major releases. Rule 1 is ok. The 2
> > minor
> > > > releases period defined in rule 2 sounds like a smooth migration
> plan,
> > > but
> > > > for users in this case, it is still not smooth.
> > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process
> with
> > > > deprecated APIs. But for those users who ignore annotation changes,
> > will
> > > > have the same issue mentioned in case 1.
> > > >
> > > > I understood that, compared to scenario 1, users, who ignore @Public
> > > > -> @Retired
> > > > changes, will not face any breaking changes, after they migrated to
> > 2.0.
> > > > But the cost of it is bridge releases. There might be many bridge
> > > releases
> > > > in the future. As I mentioned previously, it is not easy to find the
> > > right
> > > > timing to kick off the deprecation process of APIs before the next
> > > expected
> > > > major release and 2-3 years in advance.
> > > >
> > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > @deprecated
> > > > annotation changes
> > > >
> > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in
> 1.21
> > > >
> > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth
> > > > migration.
> > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > > since @deprecated is ignored
> > > >
> > > > In summary, for users who care about the annotation change, they will
> > get
> > > > more benefits with @Retired. For users who ignore annotation changes,
> > > they
> > > > will be always facing non-smooth upgrades even using pure standard
> > > > @deprecated
> > > > annotation/process.
> > > >
> > > > For the case that IDEs support @deprecated, that is true. There are
> no
> > > such
> > > > supports for homebuilt annotation. But the intention is to let users
> be
> > > > aware of it. There will be other ways to do it for users who care
> about
> > > > annotation changes. For users who do not care, there should be no
> > > > difference between with tool support or without. Using @Retired with
> > > > @deprecation
> > > > together is at least not worse than only using @deprecation alone.
> And
> > > > there are additional benefits with @Retired.
> > > >
> > > > Just my 2 cents and looking forward to your feedback, especially
> > > different
> > > > opinions that will help me understand the issue better. Thanks!
> > > >
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > <sr...@confluent.io.invalid> wrote:
> > > >
> > > > > Hi Xintong,
> > > > >
> > > > > Thanks for the summary, most of the points that you agreed upon
> also
> > > make
> > > > > sense to me.
> > > > >
> > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> demoting
> > > > APIs
> > > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > > version
> > > > > bump,
> > > > > >   are not good practices.
> > > > >
> > > > > I hope we can can go beyond calling it “not a good practice" and
> > reach
> > > > > consensus that we will not demote or remove public APIs in minor
> > > releases
> > > > > and (technically) enforce this rule for all contributions. I think
> > > it’s a
> > > > > reasonable expectation for stability from a project as mature as
> > Flink.
> > > > >
> > > > > > I'm personally in favor of option 1. As the migration-period rule
> > is
> > > > > newly
> > > > > > proposed, I think it's fair to make exceptions for cases where we
> > > > already
> > > > > > missed the best chance for planning the deprecation. Moreover, I
> do
> > > > > believe
> > > > > > having a quick major version bump does confuse users. Yes, we can
> > > > explain
> > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> other
> > > than
> > > > > the
> > > > > > removal of an deprecated API. But having to explain this itself
> is
> > an
> > > > > > indicator that it might be confusing for users.
> > > > > >
> > > > > >
> > > > > > Becket, on the other hand, prefers option 2. From my
> understanding,
> > > his
> > > > > > major point is that a quick major version bump causes barely any
> > > actual
> > > > > > lose on users, while in option 1 not providing the migration
> period
> > > or
> > > > > > providing a shorter on is an actual lose on users. (@Becket,
> please
> > > > > correct
> > > > > > me if I'm mistaken.)
> > > > > >
> > > > >
> > > > > I would be open to both options and maybe postpone this decision
> > until
> > > we
> > > > > have the complete roadmap for Flink 2.0. We would have a better
> > > overview
> > > > > about the technical consequences, e.g. how hard it would be to
> offer
> > > and
> > > > > maintain both APIs side by side, how well we get both APIs
> separated,
> > > and
> > > > > whether or not we will be able to use the full potential of our
> > > breaking
> > > > > changes before the DataStream API is completely removed. One
> example
> > > for
> > > > > the last point I can think of would be moving from lazy to eager
> > state
> > > > > declaration. If we believe that it is reasonable to support all
> > planned
> > > > > changes while maintaining both APIs, I'm leaning towards option 2.
> > > > >
> > > > > Best,
> > > > > Stefan
> > > > >
> > > > >
> > > > > > On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Becket and I had an offline voice call earlier today. We have
> > reached
> > > > > > consensus on some of the arguments, but still hold different
> > opinions
> > > > on
> > > > > > some others. I'd like to post the outcome here for transparency.
> > > > > >
> > > > > > We both agree that:
> > > > > >
> > > > > >   1. Providing a migration period would be beneficial for our
> > users,
> > > > and
> > > > > >   we should do that
> > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> demoting
> > > > APIs
> > > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > > version
> > > > > bump,
> > > > > >   are not good practices.
> > > > > >   3. Ideally, with this FLIP, developers should be more careful
> and
> > > > plan
> > > > > >   API changes ahead. That means:
> > > > > >      1. Be more careful with designing APIs and promoting them to
> > > > Public,
> > > > > >      so that they won't be changed / removed very soon, and also
> > the
> > > > > >      maintenance overhead for keeping them after deprecation
> should
> > > be
> > > > > >      affordable. (I believe this also aligns with John's
> opinion.)
> > > > > >      2. Plan the deprecation / removal of old APIs that we don't
> > want
> > > > to
> > > > > >      carry to the next major version earlier, so that they'll
> have
> > > the
> > > > > >      2-minor-release migration period before the next major
> version
> > > > bump.
> > > > > >   4. A practical situation is that, we are not ready for
> > deprecating
> > > > the
> > > > > >   DataStream API, nor afford to carry it for another couple of
> > years.
> > > > > >      1. We cannot deprecate it now, because the new
> ProcessFunction
> > > API
> > > > > >      (the planned replacement for DataStream API) is not yet
> ready.
> > > We
> > > > > haven't
> > > > > >      planned the ProcessFunction API earlier because the original
> > > > > > plan was to do
> > > > > >      in-place changes directly on the DataStream API in release
> > 2.0,
> > > > > until the
> > > > > >      smooth migration period is proposed. Before this, the only
> > > > > guarantees we
> > > > > >      provide for Public APIs is that they'll stay compatible
> until
> > > the
> > > > > next
> > > > > >      major version, which led to the understanding the in-place
> > > changes
> > > > > on
> > > > > >      Public APIs should be fine.
> > > > > >      2. We also cannot afford carrying the DataStream API for
> > another
> > > > > >      couple of years. One of the reasons that we want to
> refactor /
> > > > > replace
> > > > > >      DataStream API is that it exposes and depends on too many
> > > > > > Flink's internal
> > > > > >      runtime implementations, which has already limited / blocked
> > > plans
> > > > > for
> > > > > >      Flink's internal improvements for many times. Also due to
> the
> > > wide
> > > > > and
> > > > > >      transitive dependencies on the internals, maintaining two
> sets
> > > of
> > > > > lower
> > > > > >      APIs itself is expensive and would be better to keep it
> short.
> > > > This
> > > > > is
> > > > > >      admittedly a historical design flaw which should be avoided
> in
> > > > > future.
> > > > > >
> > > > > > What we haven't reached consensus is about how to deal with the
> > > current
> > > > > > situation around the DataStream API. There are two options.
> > > > > >
> > > > > >
> > > > > >   1. We say this FLIP is enforced starting release 2.0. For
> current
> > > 1.x
> > > > > >   APIs, we provide a migration period with best effort, while
> > > allowing
> > > > > >   exceptions for immediate removal in 2.0. That means we will
> still
> > > try
> > > > > with
> > > > > >   best effort to get the ProcessFuncion API ready and deprecate
> the
> > > > > >   DataStream API in 1.x, but will also be allowed to remove
> > > DataStream
> > > > > API in
> > > > > >   2.0 if it's not deprecated 2 minor releases before the major
> > > version
> > > > > bump.
> > > > > >   2. We strictly follow the process in this FLIP, and will
> quickly
> > > bump
> > > > > >   the major version from 2.x to 3.0 once the migration period for
> > > > > DataStream
> > > > > >   API is reached.
> > > > > >
> > > > > > I'm personally in favor of option 1. As the migration-period rule
> > is
> > > > > newly
> > > > > > proposed, I think it's fair to make exceptions for cases where we
> > > > already
> > > > > > missed the best chance for planning the deprecation. Moreover, I
> do
> > > > > believe
> > > > > > having a quick major version bump does confuse users. Yes, we can
> > > > explain
> > > > > > to users that bumping from 2.x to 3.0 does not cost anything
> other
> > > than
> > > > > the
> > > > > > removal of an deprecated API. But having to explain this itself
> is
> > an
> > > > > > indicator that it might be confusing for users.
> > > > > >
> > > > > >
> > > > > > Becket, on the other hand, prefers option 2. From my
> understanding,
> > > his
> > > > > > major point is that a quick major version bump causes barely any
> > > actual
> > > > > > lose on users, while in option 1 not providing the migration
> period
> > > or
> > > > > > providing a shorter on is an actual lose on users. (@Becket,
> please
> > > > > correct
> > > > > > me if I'm mistaken.)
> > > > > >
> > > > > >
> > > > > > And we'd like to hear more feedback from the community.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> vvcephei@apache.org
> > > > > <ma...@apache.org>> wrote:
> > > > > >
> > > > > >> Hi Becket and Xintong,
> > > > > >>
> > > > > >> I hope you don't mind if I chime in a little.
> > > > > >>
> > > > > >> Once an API is marked Public, we're committing to support it
> until
> > > > it's
> > > > > >> deprecated, and once it's deprecated, to leave it in place for
> at
> > > > least
> > > > > two
> > > > > >> minor releases and then only remove it in the following major
> > > release.
> > > > > >>
> > > > > >> As a user, I would be dismayed to discover that the project
> > > > maintainers
> > > > > >> demoted a Public API to PublicEvolving or Experimental in order
> to
> > > > > violate
> > > > > >> the spirit of the deprecation period for Public APIs. Likewise,
> > I'd
> > > > > view a
> > > > > >> rapid sequence of minor releases for the purpose of dropping
> > > > deprecated
> > > > > >> APIs in the next major release as another violation of the
> spirit
> > of
> > > > our
> > > > > >> guarantees. I'm reminded of this xkcd:
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > >>
> > > > > >> I'm glad we're gaming out these situations before we institute
> > this
> > > > > FLIP,
> > > > > >> and I hope we all converge on a clear understanding of what
> we're
> > > > > >> guaranteeing. From where I'm sitting, the essence of this FLIP
> is
> > > > > >> specifically to prevent us from doing whatever we want and
> instead
> > > to
> > > > > >> commit to providing a suitable notice period to users who rely
> on
> > > > Public
> > > > > >> APIs.
> > > > > >>
> > > > > >> It honestly shouldn't matter what the maintenance overhead is
> (or,
> > > > > rather,
> > > > > >> evaluating the maintenance overhead should be a consideration
> > before
> > > > we
> > > > > >> promote an API to Public to begin with). My experience in other
> > > > > projects is
> > > > > >> that regretting a feature so much you want to drop it asap is
> > > > extremely
> > > > > >> rare, and that living with the pain for a little while until the
> > > > > >> deprecation period expires will probably make us better
> engineers.
> > > > > >>
> > > > > >> With that all said, one thing that is not a violation is to
> > propose
> > > a
> > > > > new
> > > > > >> major release in order to drop especially burdensome deprecated
> > APIs
> > > > > that
> > > > > >> have already enjoyed their full deprecation period. Those really
> > bad
> > > > > "case
> > > > > >> A" features will be very rare.
> > > > > >>
> > > > > >> Looking over the FLIP again, I see a reference to the release
> > > > > >> documentation, but not to the build tooling support for the
> > > Deprecated
> > > > > >> annotation, so it might help to share this scenario: The way
> this
> > > will
> > > > > all
> > > > > >> come together in practice is that, by giving good stability
> > > > guarantees,
> > > > > we
> > > > > >> will encourage our users to bump minor versions when they are
> > > > released,
> > > > > >> which means that they will see compiler warnings immediately if
> > any
> > > of
> > > > > the
> > > > > >> APIs they've used are deprecated. In fact, I'd recommend for
> them
> > to
> > > > use
> > > > > >> the `-Werror` javac flag to be sure that they don't miss those
> > > > warnings.
> > > > > >> Given our release cadence, this means that they will have about
> > six
> > > > > months
> > > > > >> to migrate off of any deprecated APIs without being forced to
> > avoid
> > > > > >> updates. And they really need to be able to adopt further minor
> > > > updates
> > > > > in
> > > > > >> the series, since they may contain further improvements that
> > > actually
> > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > >>
> > > > > >> The compiler warnings are a far more reliable mechanism to
> > advertise
> > > > > >> deprecation than release notes, since users' own builds will
> > notify
> > > > them
> > > > > >> right away about only the deprecations that are relevant to
> them,
> > > but
> > > > it
> > > > > >> does rely on users feeling ok to bump up minor versions in a
> > timely
> > > > > >> fashion. If we consistently burn them by not observing the
> > stability
> > > > > >> guarantees, they'll try to avoid updates instead, and the whole
> > > thing
> > > > > falls
> > > > > >> apart.
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> -John
> > > > > >>
> > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > >>> Hi Xintong,
> > > > > >>>
> > > > > >>> Let's compare the following cases:
> > > > > >>>
> > > > > >>> A. If the maintenance overhead of the deprecated API is high,
> and
> > > we
> > > > > want
> > > > > >>> to remove it after two minor releases. Then there are two
> > options:
> > > > > >>>    A1: Demote the API in the major version bump and remove the
> > code
> > > > > >> with a
> > > > > >>> minor version bump.
> > > > > >>>    A2: Do exactly the same as 1-A, except that when removing
> the
> > > > code,
> > > > > >>> bump up the major version. So this might be a short major
> > version.
> > > > > >>>
> > > > > >>> B. If the maintenance overhead of the deprecated API is not
> high,
> > > > > >> therefore
> > > > > >>> keeping it for a long time is affordable. There are also two
> > > options:
> > > > > >>>    B1: Same as A-1, demote the API in the major version bump
> and
> > > > remove
> > > > > >>> the code with a minor version bump.
> > > > > >>>    B2: Keep the API for all the minor versions in the major
> > > version,
> > > > > and
> > > > > >>> only remove the code in the next major version.
> > > > > >>>
> > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > >>>
> > > > > >>> For case A:
> > > > > >>> The following stays the same for A1 and A2:
> > > > > >>>  - users will lose the API after two minor leases. So the
> > migration
> > > > > >> period
> > > > > >>> is the same for A-1 and A-2.
> > > > > >>>  - A1 and A2 will have exactly the same code after the removal
> of
> > > > > >>> deprecated API, the only difference is versioning.
> > > > > >>>  - To move forward, users need to move to the next minor
> version
> > in
> > > > A1,
> > > > > >> or
> > > > > >>> to the next major version in A2. Because the code are the same,
> > the
> > > > > >> actual
> > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > >>>
> > > > > >>> The differences between A-1 and A-2 are:
> > > > > >>>  - A1 allows keeping the major version release cadence. A-2
> will
> > > > have a
> > > > > >>> short major version release.
> > > > > >>>  - A1 breaks the well understood API semantic, while A-2 does
> > not.
> > > > > >>>
> > > > > >>> From what I see, since there is no well establish standard
> > > regarding
> > > > > how
> > > > > >>> long a major version should be released, A short major version
> > > > release
> > > > > is
> > > > > >>> potential and emotional. It is not ideal but does not have
> > material
> > > > > >>> downsides compared with A1. I did not hear anyone complaining
> > about
> > > > > Kafka
> > > > > >>> only has two 1.x release. However, A1 actually breaks the well
> > > > > understood
> > > > > >>> API semantic, which has more material impact.
> > > > > >>>
> > > > > >>> Also, I'd imagine 90% or more of the Public APIs should fall
> into
> > > > case
> > > > > B.
> > > > > >>> So, short major versions should be very occasional. I'd be very
> > > > > concerned
> > > > > >>> if the reason we choose A1 is simply because we cannot afford
> > > > > >> maintaining a
> > > > > >>> bunch of deprecated APIs until the next major version. This
> > > indicates
> > > > > >> that
> > > > > >>> the actual problem we need to solve is to lower the maintenance
> > > > > overhead
> > > > > >> of
> > > > > >>> deprecated APIs, so that we are comfortable to keep them
> longer.
> > As
> > > > > John
> > > > > >>> and I mentioned earlier, there are ways to achieve this and we
> > need
> > > > to
> > > > > >>> learn how to do it in Flink. Otherwise, our discussion about
> > > > versioning
> > > > > >>> here does not bring much value, because we will end up with a
> > bunch
> > > > of
> > > > > >>> short-lived APIs which upset our users, no matter how we
> version
> > > the
> > > > > >>> releases.
> > > > > >>>
> > > > > >>> So, if there are concrete examples that you think will block us
> > > from
> > > > > >>> keeping API stability with affordable cost, let's take a look
> > > > together
> > > > > >> and
> > > > > >>> see if that can be improved.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>> Jiangjie (Becket) Qin
> > > > > >>>
> > > > > >>>
> > > > > >>> From what I see,
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > tonysong820@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>>>
> > > > > >>>>> The part I don't understand is if we are willing to have a
> > > > migration
> > > > > >>>>> period, and do a minor version bump to remove an API, what do
> > we
> > > > > >> lose to
> > > > > >>>> do
> > > > > >>>>> a major version bump instead, so we don't break the common
> > > > versioning
> > > > > >>>>> semantic?
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> I think we are talking about the cases where a major version
> > bump
> > > > > >> happens
> > > > > >>>> before a deprecated Public API reaches its migration period.
> So
> > > > > >> removing
> > > > > >>>> the API with another major version bump means we have two
> > > > consecutive
> > > > > >> major
> > > > > >>>> versions in a very short time. And as previously mentioned,
> > having
> > > > > >> frequent
> > > > > >>>> major version bumps would weaken the value of the commitment
> > > "Public
> > > > > >> API
> > > > > >>>> stay compatible within a major version". I think users also
> have
> > > > > >>>> expectations about how long a major version should live, which
> > > > should
> > > > > >> be at
> > > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > > >>>>
> > > > > >>>> This is another option, but I think it is very likely more
> > > expensive
> > > > > >> than
> > > > > >>>>> simply bumping the major version. And I imagine there will be
> > > > > >> questions
> > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> 2.0"?
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> Yes, it's more expensive than simply bumping the major
> version,
> > > but
> > > > > >> IMHO
> > > > > >>>> it's probably cheaper than carrying the API until the next
> major
> > > > > >> version if
> > > > > >>>> we don't bump the major version very soon. And see my reply
> > above
> > > on
> > > > > >> why
> > > > > >>>> not bumping immediately.
> > > > > >>>>
> > > > > >>>> Best,
> > > > > >>>>
> > > > > >>>> Xintong
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > becket.qin@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > >> wrote:
> > > > > >>>>
> > > > > >>>>> Hi Xintong,
> > > > > >>>>>
> > > > > >>>>> Please see the replies below.
> > > > > >>>>>
> > > > > >>>>> I see your point, that users would feel surprised if they
> find
> > > > > >> things no
> > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > However,
> > > > > >> I'd
> > > > > >>>>> like
> > > > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > > > >> problem
> > > > > >>>>>> anyway. So the question is, how do we catch users' attention
> > and
> > > > > >> make
> > > > > >>>>> sure
> > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> > > Public
> > > > > >> in
> > > > > >>>>> 2.0.
> > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > warnings
> > > in
> > > > > >>>> logs,
> > > > > >>>>>> etc.
> > > > > >>>>>
> > > > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs
> in
> > > > minor
> > > > > >>>>> version changes. Personally speaking, I want them to be also
> > > > removed
> > > > > >> in
> > > > > >>>>> major version changes.
> > > > > >>>>>
> > > > > >>>>> Empirically, I rarely see projects with complex API semantic
> > work
> > > > > >> well.
> > > > > >>>>> Many, if not most, users don't read docs / release notes /
> > > warnings
> > > > > >> in
> > > > > >>>>> logs. I don't think that is their fault. Typically an
> > application
> > > > > >>>> developer
> > > > > >>>>> will have to deal with dozens of libraries maintained by
> > various
> > > > > >>>>> communities / groups. It is just too difficult for them to
> keep
> > > > > >> track of
> > > > > >>>>> all the specific semantics each project puts there. If you
> > think
> > > > > >> about
> > > > > >>>> it,
> > > > > >>>>> how many of Flink developers read all the release notes of
> > Guava,
> > > > > >> Apache
> > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a
> version
> > > of
> > > > > >> them?
> > > > > >>>>> One would probably try bumping the dependency version and see
> > if
> > > > > >> there is
> > > > > >>>>> an exception and solve them case by case. And if it is a
> minor
> > > > > >> version
> > > > > >>>>> bump, one probably does not expect an exception at all.
> Another
> > > > > >> example
> > > > > >>>> is
> > > > > >>>>> that in Flink we still have so many usage of deprecated
> methods
> > > all
> > > > > >> over
> > > > > >>>>> the place, and I strongly doubt everyone knows when the
> source
> > > code
> > > > > >> of
> > > > > >>>>> these methods are deprecated and when they should be removed.
> > > > > >>>>>
> > > > > >>>>> So, the most important principle of API is simple and
> > intuitive.
> > > > The
> > > > > >>>>> versioning semantic is a simple and universally accepted API
> > > > > >> stability
> > > > > >>>>> standard. If we ourselves as Flink developers are relying on
> > this
> > > > > >> for our
> > > > > >>>>> own dependencies. I don't think we can expect more from our
> > > users.
> > > > > >>>>>
> > > > > >>>>> Another possible alternative: whenever there's a deprecated
> > > Public
> > > > > >> API
> > > > > >>>> that
> > > > > >>>>>> reaches a major version bump before the migration period,
> and
> > we
> > > > > >> also
> > > > > >>>>> don't
> > > > > >>>>>> want to carry it for all the next major release series, we
> may
> > > > > >> consider
> > > > > >>>>>> releasing more minor releases for the previous major version
> > > after
> > > > > >> the
> > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> > > bump
> > > > > >> to
> > > > > >>>> 2.0,
> > > > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> > > users
> > > > > >>>> another
> > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > > >>>> 2-minor-release
> > > > > >>>>>> migration period.
> > > > > >>>>>
> > > > > >>>>> This is another option, but I think it is very likely more
> > > > expensive
> > > > > >> than
> > > > > >>>>> simply bumping the major version. And I imagine there will be
> > > > > >> questions
> > > > > >>>>> like "why this feature is in 1.20, but does not exist in
> 2.0"?
> > > > > >>>>>
> > > > > >>>>> I think my major point is, we should not carry APIs
> deprecated
> > > in a
> > > > > >>>>>> previous major version along all the next major version
> > series.
> > > > I'd
> > > > > >>>> like
> > > > > >>>>> to
> > > > > >>>>>> try giving users more commitments, i.e. the migration
> period,
> > as
> > > > > >> long
> > > > > >>>> as
> > > > > >>>>> it
> > > > > >>>>>> does not prevent us from making breaking changes. If it
> > doesn't
> > > > > >> work,
> > > > > >>>> I'd
> > > > > >>>>>> be in favor of not providing the migration period, but
> > fallback
> > > to
> > > > > >> only
> > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > >>>>>
> > > > > >>>>> The part I don't understand is if we are willing to have a
> > > > migration
> > > > > >>>>> period, and do a minor version bump to remove an API, what do
> > we
> > > > > >> lose to
> > > > > >>>> do
> > > > > >>>>> a major version bump instead, so we don't break the common
> > > > versioning
> > > > > >>>>> semantic?
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>>
> > > > > >>>>> Jiangjie (Becket) Qin
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > tonysong820@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> As an end user who only uses Public APIs, if I don't change
> > my
> > > > > >> code
> > > > > >>>> at
> > > > > >>>>>>> all, my expectation is the following:
> > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> > fine
> > > > > >> with
> > > > > >>>>> all
> > > > > >>>>>>> the 2.x versions.
> > > > > >>>>>>> Actually I think there are some dependency version
> resolution
> > > > > >>>> policies
> > > > > >>>>>> out
> > > > > >>>>>>> there which picks the highest minor version when the
> > > dependencies
> > > > > >>>> pull
> > > > > >>>>> in
> > > > > >>>>>>> multiple minor versions of the same jar, which may be
> broken
> > if
> > > > > >> we
> > > > > >>>>> remove
> > > > > >>>>>>> the API in minor releases.
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> I see your point, that users would feel surprised if they
> find
> > > > > >> things
> > > > > >>>> no
> > > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > > However,
> > > > > >> I'd
> > > > > >>>>> like
> > > > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > > > >> problem
> > > > > >>>>>> anyway. So the question is, how do we catch users' attention
> > and
> > > > > >> make
> > > > > >>>>> sure
> > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> > > Public
> > > > > >> in
> > > > > >>>>> 2.0.
> > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > warnings
> > > in
> > > > > >>>> logs,
> > > > > >>>>>> etc.
> > > > > >>>>>>
> > > > > >>>>>> Another possible alternative: whenever there's a deprecated
> > > Public
> > > > > >> API
> > > > > >>>>> that
> > > > > >>>>>> reaches a major version bump before the migration period,
> and
> > we
> > > > > >> also
> > > > > >>>>> don't
> > > > > >>>>>> want to carry it for all the next major release series, we
> may
> > > > > >> consider
> > > > > >>>>>> releasing more minor releases for the previous major version
> > > after
> > > > > >> the
> > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> > > bump
> > > > > >> to
> > > > > >>>> 2.0,
> > > > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> > > users
> > > > > >>>> another
> > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > > >>>> 2-minor-release
> > > > > >>>>>> migration period.
> > > > > >>>>>>
> > > > > >>>>>> I think my major point is, we should not carry APIs
> deprecated
> > > in
> > > > a
> > > > > >>>>>> previous major version along all the next major version
> > series.
> > > > I'd
> > > > > >>>> like
> > > > > >>>>> to
> > > > > >>>>>> try giving users more commitments, i.e. the migration
> period,
> > as
> > > > > >> long
> > > > > >>>> as
> > > > > >>>>> it
> > > > > >>>>>> does not prevent us from making breaking changes. If it
> > doesn't
> > > > > >> work,
> > > > > >>>> I'd
> > > > > >>>>>> be in favor of not providing the migration period, but
> > fallback
> > > to
> > > > > >> only
> > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > >>>>>>
> > > > > >>>>>> Best,
> > > > > >>>>>>
> > > > > >>>>>> Xintong
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > vvcephei@apache.org
> > > > > <ma...@apache.org>
> > > > > >>>
> > > > > >>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi Becket,
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks for the reply! I’d like to continue the conversation
> > > about
> > > > > >>>>>>> compatibility outside of this FLIP thread, but for now, I
> can
> > > > > >> accept
> > > > > >>>>> your
> > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks again,
> > > > > >>>>>>> John
> > > > > >>>>>>>
> > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > >>>>>>>> Hi John,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Completely agree with all you said.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Can we consider only dropping deprecated APIs in major
> > > releases
> > > > > >>>>> across
> > > > > >>>>>>> the
> > > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving
> > APIs
> > > > > >> are
> > > > > >>>> by
> > > > > >>>>>>>>> definition less stable, but it seems like this should be
> > > > > >> reflected
> > > > > >>>>> in
> > > > > >>>>>>> the
> > > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> > > them
> > > > > >>>>> around
> > > > > >>>>>>> for
> > > > > >>>>>>>>> at least zero or one minor release, not that we can drop
> > them
> > > > > >> in a
> > > > > >>>>>>> minor or
> > > > > >>>>>>>>> patch release.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Personally speaking, I would love to do this, for exactly
> > the
> > > > > >>>> reason
> > > > > >>>>>> you
> > > > > >>>>>>>> mentioned. However, I did not propose this due to the
> > > following
> > > > > >>>>>> reasons:
> > > > > >>>>>>>>
> > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> accepted
> > > > > >> FLIPs
> > > > > >>>> too
> > > > > >>>>>>> soon.
> > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> development.
> > At
> > > > > >> this
> > > > > >>>>>>> point,
> > > > > >>>>>>>> Flink still lacks some design / routines to support good
> API
> > > > > >>>>>>> evolvability /
> > > > > >>>>>>>> extensibility. Just like you said, it takes some time to
> be
> > > > > >> good at
> > > > > >>>>>> this.
> > > > > >>>>>>>> In this case, my concern is that only removing
> Experimental
> > /
> > > > > >>>>>>>> PublicEvolving APIs in major version changes may result in
> > too
> > > > > >> much
> > > > > >>>>>>>> overhead and dramatically slow down the development of
> > Flink.
> > > > > >> So, I
> > > > > >>>>> was
> > > > > >>>>>>>> thinking that we can start with the current status.
> > Hopefully
> > > > > >> after
> > > > > >>>>> we
> > > > > >>>>>>> are
> > > > > >>>>>>>> more comfortable with the maintenance overhead of
> deprecated
> > > > > >> APIs,
> > > > > >>>> we
> > > > > >>>>>> can
> > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > >> PublicEvolving
> > > > > >>>>> APIs.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi Becket,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is
> > really
> > > > > >>>>>> important.
> > > > > >>>>>>> I
> > > > > >>>>>>>>> understand some people’s concerns about the additional
> > burden
> > > > > >> for
> > > > > >>>>>>> project
> > > > > >>>>>>>>> maintainers, but my personal experience with Kafka has
> been
> > > > > >> that
> > > > > >>>>> it’s
> > > > > >>>>>>> very
> > > > > >>>>>>>>> liveable and that it’s well worth the benefit to users.
> In
> > > > > >> fact,
> > > > > >>>>> users
> > > > > >>>>>>>>> being able to confidently upgrade is also a benefit to
> > > > > >>>> maintainers,
> > > > > >>>>> as
> > > > > >>>>>>> we
> > > > > >>>>>>>>> will get fewer questions from people stuck on very old
> > > > > >> versions.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> One question:
> > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in major
> > > > > >> releases
> > > > > >>>>> across
> > > > > >>>>>>> the
> > > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving
> > APIs
> > > > > >> are
> > > > > >>>> by
> > > > > >>>>>>>>> definition less stable, but it seems like this should be
> > > > > >> reflected
> > > > > >>>>> in
> > > > > >>>>>>> the
> > > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> > > them
> > > > > >>>>> around
> > > > > >>>>>>> for
> > > > > >>>>>>>>> at least zero or one minor release, not that we can drop
> > them
> > > > > >> in a
> > > > > >>>>>>> minor or
> > > > > >>>>>>>>> patch release.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The advantage of forbidding the removal of any API in
> minor
> > > or
> > > > > >>>> patch
> > > > > >>>>>>>>> releases is that users will get a strong guarantee that
> > they
> > > > > >> can
> > > > > >>>>> bump
> > > > > >>>>>>> the
> > > > > >>>>>>>>> minor or patch version and still be able to compile, or
> > even
> > > > > >> just
> > > > > >>>>>>> re-link
> > > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at
> run
> > > > > >> time.
> > > > > >>>>> This
> > > > > >>>>>>> is a
> > > > > >>>>>>>>> binary guarantee: if we allow removing  even Experimental
> > > APIs
> > > > > >>>>> outside
> > > > > >>>>>>> of
> > > > > >>>>>>>>> major releases, users can no longer confidently upgrade.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of
> > points:
> > > > > >>>>>>>>> * I’d use the official Deprecated annotation instead of
> > > > > >>>> introducing
> > > > > >>>>>> our
> > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > > integrated
> > > > > >>>> into
> > > > > >>>>>>> build
> > > > > >>>>>>>>> tools and IDEs.
> > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP;
> > it
> > > > > >> seems
> > > > > >>>>>>>>> orthogonal, and something that should probably be taken
> > > > > >>>> case-by-case
> > > > > >>>>>>>>> anyway.
> > > > > >>>>>>>>> * Aside from deprecation and removal, there have been
> some
> > > > > >>>>> discussions
> > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways.
> > > > > >> This is
> > > > > >>>>>>> somewhat
> > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before,
> > > it’ll
> > > > > >>>> take
> > > > > >>>>>> some
> > > > > >>>>>>>>> time to become good at it. I feel like this topic should
> > also
> > > > > >> be
> > > > > >>>>>>> orthogonal
> > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
> > > > > >> simple
> > > > > >>>>>> policy
> > > > > >>>>>>> not
> > > > > >>>>>>>>> to break existing user programs, and leave the “how” up
> to
> > > > > >>>>>> implementers
> > > > > >>>>>>> and
> > > > > >>>>>>>>> reviewers.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks again,
> > > > > >>>>>>>>> John
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > >>>>>>>>>> Hi All,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
> > > > > >> great
> > > > > >>>>> idea.
> > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > > >>>> @PublicEvolving
> > > > > >>>>>> --(2
> > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > > > > >> flexibility
> > > > > >>>>>>> without
> > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> > > > > >> change
> > > > > >>>>>> between
> > > > > >>>>>>>>> major
> > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> provides
> > > > > >>>>> additional
> > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> themself
> > > > > >> are
> > > > > >>>> not
> > > > > >>>>>>>>> changed.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I
> > > > > >> want to
> > > > > >>>>>> move
> > > > > >>>>>>> one
> > > > > >>>>>>>>>> step further and suggest introducing one more annotation
> > > > > >>>> @Retired.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > > > > >> downgrading
> > > > > >>>>>>> @Public
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply,
> Java
> > > > > >>>> standard
> > > > > >>>>>>>>>> @deprecated should be used in the early stage of the
> > > > > >> deprecation
> > > > > >>>>>>> process
> > > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does
> > not
> > > > > >>>> allow
> > > > > >>>>>> us
> > > > > >>>>>>> to
> > > > > >>>>>>>>>> extend annotation, I think it would be feasible to have
> > the
> > > > > >> new
> > > > > >>>>>>> @Retired
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>> help us monitor and manage the deprecation process,
> house
> > > > > >>>>> cleaning,
> > > > > >>>>>>> etc.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> @Retired:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > > >> compatibility
> > > > > >>>>>> before
> > > > > >>>>>>>>> APIs
> > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> processes
> > > > > >> on
> > > > > >>>> the
> > > > > >>>>> ML
> > > > > >>>>>>> are
> > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
> > > > > >> release
> > > > > >>>>>>> sharply
> > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether
> > it
> > > > > >> is
> > > > > >>>>>>> possible).
> > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to
> remove
> > > > > >> them.
> > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> > @Retired
> > > > > >> are
> > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
> > > > > >> @Retired
> > > > > >>>>>> could
> > > > > >>>>>>>>> also
> > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > >>>> dev-friendliness,
> > > > > >>>>>> but
> > > > > >>>>>>> not
> > > > > >>>>>>>>>> mandatory.
> > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support
> > the
> > > > > >>>>>>> deprecation
> > > > > >>>>>>>>>> process management. Further extension is possible, since
> > the
> > > > > >>>>>>> annotation
> > > > > >>>>>>>>> is
> > > > > >>>>>>>>>> built by us.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best regards,
> > > > > >>>>>>>>>> Jing
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > > >>>>>>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Hi Xintong,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies
> inline
> > > > > >>>> below.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public
> API
> > > > > >> is
> > > > > >>>>> also a
> > > > > >>>>>>>>> kind of
> > > > > >>>>>>>>>>>> such change, just like removing one, which can only
> > > > > >> happen
> > > > > >>>> with
> > > > > >>>>>>> major
> > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > Public
> > > > > >>>> APIs
> > > > > >>>>>>>>> anytime,
> > > > > >>>>>>>>>>> but
> > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > >> reaching
> > > > > >>>> the
> > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > > >> would
> > > > > >>>> be a
> > > > > >>>>>>> weaker
> > > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > > >> major
> > > > > >>>>>> version
> > > > > >>>>>>>>> bumps,
> > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > 2-minor-release
> > > > > >>>>>> migration
> > > > > >>>>>>>>>>> period.
> > > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > > > >> sounds
> > > > > >>>>>>> against
> > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > >> necessary,
> > > > > >>>>>> e.g.,
> > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> > > > > >> change my
> > > > > >>>>> code
> > > > > >>>>>>> at
> > > > > >>>>>>>>> all,
> > > > > >>>>>>>>>>> my expectation is the following:
> > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I
> > am
> > > > > >> fine
> > > > > >>>>>> with
> > > > > >>>>>>> all
> > > > > >>>>>>>>>>> the 2.x versions.
> > > > > >>>>>>>>>>> Actually I think there are some dependency version
> > > > > >> resolution
> > > > > >>>>>>> policies
> > > > > >>>>>>>>> out
> > > > > >>>>>>>>>>> there which picks the highest minor version when the
> > > > > >>>> dependencies
> > > > > >>>>>>> pull
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>> multiple minor versions of the same jar, which may be
> > > > > >> broken if
> > > > > >>>>> we
> > > > > >>>>>>>>> remove
> > > > > >>>>>>>>>>> the API in minor releases.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > that
> > > > > >> we
> > > > > >>>>> bump
> > > > > >>>>>> up
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > > >> However,
> > > > > >>>>> this
> > > > > >>>>>>> also
> > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> > will
> > > > > >>>> stay
> > > > > >>>>>>> stable
> > > > > >>>>>>>>>>>> within the major release series, as the series can end
> > > > > >>>> anytime.
> > > > > >>>>>>> IMHO,
> > > > > >>>>>>>>>>> short
> > > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > > >> users
> > > > > >>>>>> happy",
> > > > > >>>>>>> but
> > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > >> frequent
> > > > > >>>>>> breaking
> > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach,
> we
> > > > > >> can
> > > > > >>>>>> still
> > > > > >>>>>>>>> have
> > > > > >>>>>>>>>>>> longer major release series, while only allowing
> Public
> > > > > >> APIs
> > > > > >>>>>>>>> deprecated
> > > > > >>>>>>>>>>> at
> > > > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > > > >> the
> > > > > >>>> next
> > > > > >>>>>>> major
> > > > > >>>>>>>>>>>> version.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I totally agree that frequent major version bumps are
> not
> > > > > >>>> ideal,
> > > > > >>>>>> but
> > > > > >>>>>>>>> here
> > > > > >>>>>>>>>>> we are comparing it with a minor version bump which
> > > > > >> removes a
> > > > > >>>>>> Public
> > > > > >>>>>>>>> API.
> > > > > >>>>>>>>>>> So the context is that we have already decided to
> remove
> > > > > >> this
> > > > > >>>>>> Public
> > > > > >>>>>>> API
> > > > > >>>>>>>>>>> while keeping everything else backwards compatible. I
> > > > > >> think a
> > > > > >>>>> major
> > > > > >>>>>>>>> version
> > > > > >>>>>>>>>>> bump is a commonly understood signal here, compared
> with
> > a
> > > > > >>>> minor
> > > > > >>>>>>> version
> > > > > >>>>>>>>>>> change. From end users' perspective, for those who are
> > not
> > > > > >>>>>> impacted,
> > > > > >>>>>>> in
> > > > > >>>>>>>>>>> this case upgrading a major version is not necessarily
> > more
> > > > > >>>>>> involved
> > > > > >>>>>>>>> than
> > > > > >>>>>>>>>>> upgrading a minor version - both should be as smooth
> as a
> > > > > >>>>>> dependency
> > > > > >>>>>>>>>>> version change. For those who are impacted, they will
> > lose
> > > > > >> the
> > > > > >>>>>> Public
> > > > > >>>>>>>>> API
> > > > > >>>>>>>>>>> anyways and a major version bump ensures there is no
> > > > > >> surprise.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > > > >>>>>>>>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Public API is a well defined common concept, and one
> of
> > > > > >> its
> > > > > >>>>>>>>>>>>> convention is that it only changes with a major
> version
> > > > > >>>>> change.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public
> > > > > >> API is
> > > > > >>>>>> also a
> > > > > >>>>>>>>> kind
> > > > > >>>>>>>>>>>> of such change, just like removing one, which can only
> > > > > >> happen
> > > > > >>>>>> with
> > > > > >>>>>>>>> major
> > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> > Public
> > > > > >>>> APIs
> > > > > >>>>>>>>> anytime,
> > > > > >>>>>>>>>>> but
> > > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > > >> reaching
> > > > > >>>> the
> > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > > >> would
> > > > > >>>> be a
> > > > > >>>>>>> weaker
> > > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > > >> major
> > > > > >>>>>> version
> > > > > >>>>>>>>> bumps,
> > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > 2-minor-release
> > > > > >>>>>> migration
> > > > > >>>>>>>>>>> period.
> > > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > > > >> sounds
> > > > > >>>>>>> against
> > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > > >> necessary,
> > > > > >>>>>> e.g.,
> > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> But it should be
> > > > > >>>>>>>>>>>>> completely OK to bump up the major version if we
> really
> > > > > >> want
> > > > > >>>>> to
> > > > > >>>>>>> get
> > > > > >>>>>>>>> rid
> > > > > >>>>>>>>>>> of
> > > > > >>>>>>>>>>>>> a public API, right?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > > > > >> that we
> > > > > >>>>>> bump
> > > > > >>>>>>> up
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > > >> However,
> > > > > >>>>> this
> > > > > >>>>>>> also
> > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> > will
> > > > > >>>> stay
> > > > > >>>>>>> stable
> > > > > >>>>>>>>>>>> within the major release series, as the series can end
> > > > > >>>> anytime.
> > > > > >>>>>>> IMHO,
> > > > > >>>>>>>>>>> short
> > > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > > >> users
> > > > > >>>>>> happy",
> > > > > >>>>>>> but
> > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > > >> frequent
> > > > > >>>>>> breaking
> > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach,
> we
> > > > > >> can
> > > > > >>>>>> still
> > > > > >>>>>>>>> have
> > > > > >>>>>>>>>>>> longer major release series, while only allowing
> Public
> > > > > >> APIs
> > > > > >>>>>>>>> deprecated
> > > > > >>>>>>>>>>> at
> > > > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > > > >> the
> > > > > >>>> next
> > > > > >>>>>>> major
> > > > > >>>>>>>>>>>> version.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > > > >> (1-2
> > > > > >>>>> years)
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put
> > it
> > > > > >>>> again
> > > > > >>>>>> to
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation
> > on
> > > > > >>>> when
> > > > > >>>>>>>>> breaking
> > > > > >>>>>>>>>>>>> changes can be made.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
> > > > > >> for new
> > > > > >>>>>> major
> > > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > > > > >> needed.
> > > > > >>>> That
> > > > > >>>>>>> makes
> > > > > >>>>>>>>> 1-2
> > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while
> leaving
> > > > > >> us
> > > > > >>>> more
> > > > > >>>>>>> time
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Xintong
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
> > > > > >>>>> versions
> > > > > >>>>>>> for
> > > > > >>>>>>>>>>>>> previous
> > > > > >>>>>>>>>>>>>> major versions.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> > > > > >> viable
> > > > > >>>> that
> > > > > >>>>>> is
> > > > > >>>>>>> in
> > > > > >>>>>>>>>>>>> practice.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > > > > >> support
> > > > > >>>> for
> > > > > >>>>>> Java
> > > > > >>>>>>>>> 8/11,
> > > > > >>>>>>>>>>>>> which may very well be a problem for our current
> users.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
> > > > > >> about
> > > > > >>>>>> Public
> > > > > >>>>>>>>> API
> > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
> > > > > >> in
> > > > > >>>> the
> > > > > >>>>>>> NOTES
> > > > > >>>>>>>>> part
> > > > > >>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
> > > > > >> it.
> > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
> > > > > >> are on
> > > > > >>>>> the
> > > > > >>>>>>> same
> > > > > >>>>>>>>>>>>> page on that note.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > >>>>>>>>>>>>>> But it should be
> > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > > > > >> really
> > > > > >>>> want
> > > > > >>>>>> to
> > > > > >>>>>>> get
> > > > > >>>>>>>>>>> rid
> > > > > >>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get
> us
> > > > > >> to
> > > > > >>>>> 2.0.
> > > > > >>>>>> ;)
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the
> cadence
> > > > > >> of
> > > > > >>>>> major
> > > > > >>>>>>>>> releases
> > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> opinions
> > > > > >> on
> > > > > >>>>> that.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases
> > between
> > > > > >>>> major
> > > > > >>>>>>> ones,
> > > > > >>>>>>>>>>>>> that for us means that users have to potentially deal
> > > > > >> with
> > > > > >>>>>>> breaking
> > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > > > >> (1-2
> > > > > >>>>>> years)
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put
> > it
> > > > > >>>> again
> > > > > >>>>>> to
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation
> > on
> > > > > >>>> when
> > > > > >>>>>>>>> breaking
> > > > > >>>>>>>>>>>>> changes can be made.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
> > > > > >> Not
> > > > > >>>>> sure
> > > > > >>>>>> I
> > > > > >>>>>>>>> fully
> > > > > >>>>>>>>>>>>>> understand the concern for behavioral change here.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> This could be a lot of things. It can be performance
> in
> > > > > >>>>> certain
> > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
> > > > > >> relied
> > > > > >>>>> upon
> > > > > >>>>>>>>>>>>> (
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > ),
> > > > > a semantic change to some
> > > > > >> API.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> For a concrete example, consider the job submission.
> A
> > > > > >> few
> > > > > >>>>>>> releases
> > > > > >>>>>>>>> back
> > > > > >>>>>>>>>>>>> we made changes such that the initialization of the
> job
> > > > > >>>> master
> > > > > >>>>>>>>> happens
> > > > > >>>>>>>>>>>>> asynchronously.
> > > > > >>>>>>>>>>>>> This meant the job submission call returns sooner,
> and
> > > > > >> the
> > > > > >>>> job
> > > > > >>>>>>> state
> > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but
> the
> > > > > >>>>> observed
> > > > > >>>>>>>>> behavior
> > > > > >>>>>>>>>>>>> may be different.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Metrics are another example; I believe over time we
> > > > > >> changed
> > > > > >>>>> what
> > > > > >>>>>>> some
> > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Jingsong,

Thanks for the reply. I completely agree with you.

The above 2 options are based on the assumption that the community cannot
afford to maintain the deprecated DataStream API for long. I'd say we
should try everything we can to maintain it for as much time as possible.
DataStream API is actually the most used API in Flink by so many users at
this point. Removing it any time soon will dramatically hurt our users. So
ideally we should keep it for at least 2 years after deprecation, if not
more.

The prohibitively high maintenance overhead is just an assumption.
Personally speaking, I don't feel this assumption is necessarily true. We
should re-evaluate once we have the new ProcessFunction API in place.
Without the code it is hard to tell for sure. I am actually kind of
optimistic about the maintenance cost.

Thanks,

Jiangjie (Becket) Qin



On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <ji...@gmail.com> wrote:

> Thanks Becket and all for your discussion.
>
> > 1. We say this FLIP is enforced starting release 2.0. For current 1.x
> APIs,
> we provide a migration period with best effort, while allowing exceptions
> for immediate removal in 2.0. That means we will still try with best effort
> to get the ProcessFuncion API ready and deprecate the DataStream API in
> 1.x, but will also be allowed to remove DataStream API in 2.0 if it's not
> deprecated 2 minor releases before the major version bump.
>
> > 2. We strictly follow the process in this FLIP, and will quickly bump the
> major version from 2.x to 3.0 once the migration period for DataStream API
> is reached.
>
> Sorry, I didn't read the previous detailed discussion because the
> discussion list was so long.
>
> I don't really like either of these options.
>
> Considering that DataStream is such an important API, can we offer a third
> option:
>
> 3. Maintain the DataStream API throughout 2.X and remove it until 3.x. But
> there's no need to assume that 2.X is a short version, it's still a normal
> major version.
>
> Best,
> Jingsong
>
> Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:
>
> > Thanks much for the input, John, Stefan and Jing.
> >
> > I think Xingtong has well summarized the pros and cons of the two
> options.
> > Let's collect a few more opinions here and we can move forward with the
> one
> > more people prefer.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks Xingtong for the summary. If I could only choose one of the
> given
> > > two options, I would go with option 1. I understood that option 2
> worked
> > > great with Kafka. But the bridge release will still confuse users and
> my
> > > gut feeling is that many users will skip 2.0 and be waiting for 3.0 or
> > even
> > > 3.x. And since fewer users will use Flink 2.x, the development focus
> will
> > > be on Flink 3.0 with the fact that the current Flink release is 1.17
> and
> > we
> > > are preparing 2.0 release. That is weird for me.
> > >
> > > THB, I would not name the change from @Public to @Retired as a
> demotion.
> > > The purpose of @Retire is to extend the API lifecycle with one more
> > stage,
> > > like in the real world, people born, studied, graduated, worked, and
> > > retired. Afaiu from the previous discussion, there are two rules we'd
> > like
> > > to follow simultaneously:
> > >
> > > 1. Public APIs can only be changed between major releases.
> > > 2. A smooth migration phase should be offered to users, i.e. at least 2
> > > minor releases after APIs are marked as @deprecated. There should be
> new
> > > APIs as the replacement.
> > >
> > > Agree, those rules are good to improve the user friendliness. Issues we
> > > discussed are rising because we want to fulfill both of them. If we
> take
> > > care of deprecation very seriously, APIs can be marked as @Deprecated,
> > only
> > > when the new APIs as the replacement provide all functionalities the
> > > deprecated APIs have. In an ideal case without critical bugs that might
> > > stop users adopting the new APIs. Otherwise the expected "replacement"
> > will
> > > not happen. Users will still stick to the deprecated APIs, because the
> > new
> > > APIs can not be used. For big features, it will need at least 4 minor
> > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > >
> > > - 1st minor release to build the new APIs as the replacement and
> waiting
> > > for feedback. It might be difficult to mark the old API as deprecated
> in
> > > this release, because we are not sure if the new APIs could cover 100%
> > > functionalities.
> > > -  In the lucky case,  mark all old APIs as deprecated in the 2nd minor
> > > release. (I would even suggest having the new APIs released at least
> for
> > > two minor releases before marking it as deprecated to make sure they
> can
> > > really replace the old APIs, in case we care more about smooth
> migration)
> > > - 3rd minor release for the migration period
> > > -  In another lucky case, the 4th release is a major release, the
> > > deprecated APIs could be removed.
> > >
> > > The above described scenario works only in an ideal case. In reality,
> it
> > > might take longer to get the new APIs ready and mark the old API
> > > deprecated. Furthermore, if the 4th release is not a major release, we
> > will
> > > have to maintain both APIs for many further minor releases. The
> question
> > is
> > > how to know the next major release in advance, especially 4 minor
> > releases'
> > > period, i.e. more than 2 years in advance? Given that Flink contains
> many
> > > modules, it is difficult to ask devs to create a 2-3 years deprecation
> > plan
> > > for each case. In case we want to build major releases at a fast pace,
> > > let's say every two years, it means devs must plan any API deprecation
> > > right after each major release. Afaiac, it is quite difficult.
> > >
> > > The major issue is, afaiu, if we follow rule 2, we have to keep all
> > @Public
> > > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0.
> > Then
> > > we have to follow rule 1 to keep it unchanged until we have 3.0. That
> is
> > > why @Retired is useful to give devs more flexibility and still fulfill
> > both
> > > rules. Let's check it with examples:
> > >
> > > - we have @Public DataStream API in 1.18. It will not be marked
> > > as @Deprecated, because the new APIs as the replacement are not ready.
> > > - we keep the DataStream API itself unchanged in 2.0, but change the
> > > annotation from @Public to @Retire. New APIs will be introduced too. In
> > > this case, Rule 1 is ok, since public API is allowed to change between
> > > major releases. Only changing annotation is the minimal change we could
> > do
> > > and it does not break rule 1. Rule 2 is ok too, since the DataStream
> APIs
> > > work exactly the same.  Attention: the change of @Public -> @Retired
> can
> > > only be done between major releases, because @Public APIs can only be
> > > changed between major releases.
> > > - in 2.1, DataStream API will be marked as deprecated.
> > > - in 2.2, DataStream will be kept for the migration period.
> > > - in 2.3, DataStream will be removed.
> > >
> > > Becket mentioned previously (please correct me if I didn't understand
> it
> > > correctly) that users might not check the changes of annotation and the
> > > upgrade process might not be smooth. I was wondering, if users don't
> pay
> > > attention to that, they will also not pay attention to @deprecated. The
> > > migration will not be smooth too even when we stick to the Java
> > > standard @deprecated. Let's go throw it with examples:
> > >
> > > Scenario 1: with @Retired
> > >
> > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be
> done
> > > for users who still want to use DataStream. The migration is smooth.
> But
> > > they should be aware of the annotation changes, i.e. @Public ->
> @Retired
> > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since
> it
> > is
> > > a major release upgrade, breaking changes are expected.
> > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed. Since
> > APIs
> > > are marked as @Retired, breaking changes are expected too.
> > >
> > > In addition to fulfilling both rules, there are two more benefits of
> this
> > > option:
> > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > 2. no bridge release is required. DataStream API could be deprecated in
> > 2.x
> > > without breaking any of the two rules. @Retired provides enough
> > flexibility
> > > for future deprecations and upgrades.
> > >
> > > The only issue I can find is that users might be surprised in case 3,
> if
> > > they ignore the previous @Public -> @Retired change. But for such
> users,
> > > they will ignore @deprecated annotation too and will have issues with
> > > pure @deprecated
> > > annotation too:
> > >
> > > Scenario 2: with @deprecated
> > >
> > > The example will be slightly different from scenario 1, because @Public
> > > APIs can only be removed between major releases. As an example,
> > DataStream
> > > API will be marked as deprecated in 1.19, kept in 1.20, and then
> removed
> > in
> > > 2.0
> > >
> > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as
> > > deprecated are removed between major releases. Rule 1 is ok. The 2
> minor
> > > releases period defined in rule 2 sounds like a smooth migration plan,
> > but
> > > for users in this case, it is still not smooth.
> > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with
> > > deprecated APIs. But for those users who ignore annotation changes,
> will
> > > have the same issue mentioned in case 1.
> > >
> > > I understood that, compared to scenario 1, users, who ignore @Public
> > > -> @Retired
> > > changes, will not face any breaking changes, after they migrated to
> 2.0.
> > > But the cost of it is bridge releases. There might be many bridge
> > releases
> > > in the future. As I mentioned previously, it is not easy to find the
> > right
> > > timing to kick off the deprecation process of APIs before the next
> > expected
> > > major release and 2-3 years in advance.
> > >
> > > Scenario 3: @PublicEvolving deprecation for users who ignore
> @deprecated
> > > annotation changes
> > >
> > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21
> > >
> > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth
> > > migration.
> > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > since @deprecated is ignored
> > >
> > > In summary, for users who care about the annotation change, they will
> get
> > > more benefits with @Retired. For users who ignore annotation changes,
> > they
> > > will be always facing non-smooth upgrades even using pure standard
> > > @deprecated
> > > annotation/process.
> > >
> > > For the case that IDEs support @deprecated, that is true. There are no
> > such
> > > supports for homebuilt annotation. But the intention is to let users be
> > > aware of it. There will be other ways to do it for users who care about
> > > annotation changes. For users who do not care, there should be no
> > > difference between with tool support or without. Using @Retired with
> > > @deprecation
> > > together is at least not worse than only using @deprecation alone. And
> > > there are additional benefits with @Retired.
> > >
> > > Just my 2 cents and looking forward to your feedback, especially
> > different
> > > opinions that will help me understand the issue better. Thanks!
> > >
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > <sr...@confluent.io.invalid> wrote:
> > >
> > > > Hi Xintong,
> > > >
> > > > Thanks for the summary, most of the points that you agreed upon also
> > make
> > > > sense to me.
> > > >
> > > > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> > > APIs
> > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > version
> > > > bump,
> > > > >   are not good practices.
> > > >
> > > > I hope we can can go beyond calling it “not a good practice" and
> reach
> > > > consensus that we will not demote or remove public APIs in minor
> > releases
> > > > and (technically) enforce this rule for all contributions. I think
> > it’s a
> > > > reasonable expectation for stability from a project as mature as
> Flink.
> > > >
> > > > > I'm personally in favor of option 1. As the migration-period rule
> is
> > > > newly
> > > > > proposed, I think it's fair to make exceptions for cases where we
> > > already
> > > > > missed the best chance for planning the deprecation. Moreover, I do
> > > > believe
> > > > > having a quick major version bump does confuse users. Yes, we can
> > > explain
> > > > > to users that bumping from 2.x to 3.0 does not cost anything other
> > than
> > > > the
> > > > > removal of an deprecated API. But having to explain this itself is
> an
> > > > > indicator that it might be confusing for users.
> > > > >
> > > > >
> > > > > Becket, on the other hand, prefers option 2. From my understanding,
> > his
> > > > > major point is that a quick major version bump causes barely any
> > actual
> > > > > lose on users, while in option 1 not providing the migration period
> > or
> > > > > providing a shorter on is an actual lose on users. (@Becket, please
> > > > correct
> > > > > me if I'm mistaken.)
> > > > >
> > > >
> > > > I would be open to both options and maybe postpone this decision
> until
> > we
> > > > have the complete roadmap for Flink 2.0. We would have a better
> > overview
> > > > about the technical consequences, e.g. how hard it would be to offer
> > and
> > > > maintain both APIs side by side, how well we get both APIs separated,
> > and
> > > > whether or not we will be able to use the full potential of our
> > breaking
> > > > changes before the DataStream API is completely removed. One example
> > for
> > > > the last point I can think of would be moving from lazy to eager
> state
> > > > declaration. If we believe that it is reasonable to support all
> planned
> > > > changes while maintaining both APIs, I'm leaning towards option 2.
> > > >
> > > > Best,
> > > > Stefan
> > > >
> > > >
> > > > > On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com>
> > wrote:
> > > > >
> > > > > Becket and I had an offline voice call earlier today. We have
> reached
> > > > > consensus on some of the arguments, but still hold different
> opinions
> > > on
> > > > > some others. I'd like to post the outcome here for transparency.
> > > > >
> > > > > We both agree that:
> > > > >
> > > > >   1. Providing a migration period would be beneficial for our
> users,
> > > and
> > > > >   we should do that
> > > > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> > > APIs
> > > > >   from Public to PublicEvolving / Experimental / Retired in any
> > version
> > > > bump,
> > > > >   are not good practices.
> > > > >   3. Ideally, with this FLIP, developers should be more careful and
> > > plan
> > > > >   API changes ahead. That means:
> > > > >      1. Be more careful with designing APIs and promoting them to
> > > Public,
> > > > >      so that they won't be changed / removed very soon, and also
> the
> > > > >      maintenance overhead for keeping them after deprecation should
> > be
> > > > >      affordable. (I believe this also aligns with John's opinion.)
> > > > >      2. Plan the deprecation / removal of old APIs that we don't
> want
> > > to
> > > > >      carry to the next major version earlier, so that they'll have
> > the
> > > > >      2-minor-release migration period before the next major version
> > > bump.
> > > > >   4. A practical situation is that, we are not ready for
> deprecating
> > > the
> > > > >   DataStream API, nor afford to carry it for another couple of
> years.
> > > > >      1. We cannot deprecate it now, because the new ProcessFunction
> > API
> > > > >      (the planned replacement for DataStream API) is not yet ready.
> > We
> > > > haven't
> > > > >      planned the ProcessFunction API earlier because the original
> > > > > plan was to do
> > > > >      in-place changes directly on the DataStream API in release
> 2.0,
> > > > until the
> > > > >      smooth migration period is proposed. Before this, the only
> > > > guarantees we
> > > > >      provide for Public APIs is that they'll stay compatible until
> > the
> > > > next
> > > > >      major version, which led to the understanding the in-place
> > changes
> > > > on
> > > > >      Public APIs should be fine.
> > > > >      2. We also cannot afford carrying the DataStream API for
> another
> > > > >      couple of years. One of the reasons that we want to refactor /
> > > > replace
> > > > >      DataStream API is that it exposes and depends on too many
> > > > > Flink's internal
> > > > >      runtime implementations, which has already limited / blocked
> > plans
> > > > for
> > > > >      Flink's internal improvements for many times. Also due to the
> > wide
> > > > and
> > > > >      transitive dependencies on the internals, maintaining two sets
> > of
> > > > lower
> > > > >      APIs itself is expensive and would be better to keep it short.
> > > This
> > > > is
> > > > >      admittedly a historical design flaw which should be avoided in
> > > > future.
> > > > >
> > > > > What we haven't reached consensus is about how to deal with the
> > current
> > > > > situation around the DataStream API. There are two options.
> > > > >
> > > > >
> > > > >   1. We say this FLIP is enforced starting release 2.0. For current
> > 1.x
> > > > >   APIs, we provide a migration period with best effort, while
> > allowing
> > > > >   exceptions for immediate removal in 2.0. That means we will still
> > try
> > > > with
> > > > >   best effort to get the ProcessFuncion API ready and deprecate the
> > > > >   DataStream API in 1.x, but will also be allowed to remove
> > DataStream
> > > > API in
> > > > >   2.0 if it's not deprecated 2 minor releases before the major
> > version
> > > > bump.
> > > > >   2. We strictly follow the process in this FLIP, and will quickly
> > bump
> > > > >   the major version from 2.x to 3.0 once the migration period for
> > > > DataStream
> > > > >   API is reached.
> > > > >
> > > > > I'm personally in favor of option 1. As the migration-period rule
> is
> > > > newly
> > > > > proposed, I think it's fair to make exceptions for cases where we
> > > already
> > > > > missed the best chance for planning the deprecation. Moreover, I do
> > > > believe
> > > > > having a quick major version bump does confuse users. Yes, we can
> > > explain
> > > > > to users that bumping from 2.x to 3.0 does not cost anything other
> > than
> > > > the
> > > > > removal of an deprecated API. But having to explain this itself is
> an
> > > > > indicator that it might be confusing for users.
> > > > >
> > > > >
> > > > > Becket, on the other hand, prefers option 2. From my understanding,
> > his
> > > > > major point is that a quick major version bump causes barely any
> > actual
> > > > > lose on users, while in option 1 not providing the migration period
> > or
> > > > > providing a shorter on is an actual lose on users. (@Becket, please
> > > > correct
> > > > > me if I'm mistaken.)
> > > > >
> > > > >
> > > > > And we'd like to hear more feedback from the community.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcephei@apache.org
> > > > <ma...@apache.org>> wrote:
> > > > >
> > > > >> Hi Becket and Xintong,
> > > > >>
> > > > >> I hope you don't mind if I chime in a little.
> > > > >>
> > > > >> Once an API is marked Public, we're committing to support it until
> > > it's
> > > > >> deprecated, and once it's deprecated, to leave it in place for at
> > > least
> > > > two
> > > > >> minor releases and then only remove it in the following major
> > release.
> > > > >>
> > > > >> As a user, I would be dismayed to discover that the project
> > > maintainers
> > > > >> demoted a Public API to PublicEvolving or Experimental in order to
> > > > violate
> > > > >> the spirit of the deprecation period for Public APIs. Likewise,
> I'd
> > > > view a
> > > > >> rapid sequence of minor releases for the purpose of dropping
> > > deprecated
> > > > >> APIs in the next major release as another violation of the spirit
> of
> > > our
> > > > >> guarantees. I'm reminded of this xkcd:
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > >>
> > > > >> I'm glad we're gaming out these situations before we institute
> this
> > > > FLIP,
> > > > >> and I hope we all converge on a clear understanding of what we're
> > > > >> guaranteeing. From where I'm sitting, the essence of this FLIP is
> > > > >> specifically to prevent us from doing whatever we want and instead
> > to
> > > > >> commit to providing a suitable notice period to users who rely on
> > > Public
> > > > >> APIs.
> > > > >>
> > > > >> It honestly shouldn't matter what the maintenance overhead is (or,
> > > > rather,
> > > > >> evaluating the maintenance overhead should be a consideration
> before
> > > we
> > > > >> promote an API to Public to begin with). My experience in other
> > > > projects is
> > > > >> that regretting a feature so much you want to drop it asap is
> > > extremely
> > > > >> rare, and that living with the pain for a little while until the
> > > > >> deprecation period expires will probably make us better engineers.
> > > > >>
> > > > >> With that all said, one thing that is not a violation is to
> propose
> > a
> > > > new
> > > > >> major release in order to drop especially burdensome deprecated
> APIs
> > > > that
> > > > >> have already enjoyed their full deprecation period. Those really
> bad
> > > > "case
> > > > >> A" features will be very rare.
> > > > >>
> > > > >> Looking over the FLIP again, I see a reference to the release
> > > > >> documentation, but not to the build tooling support for the
> > Deprecated
> > > > >> annotation, so it might help to share this scenario: The way this
> > will
> > > > all
> > > > >> come together in practice is that, by giving good stability
> > > guarantees,
> > > > we
> > > > >> will encourage our users to bump minor versions when they are
> > > released,
> > > > >> which means that they will see compiler warnings immediately if
> any
> > of
> > > > the
> > > > >> APIs they've used are deprecated. In fact, I'd recommend for them
> to
> > > use
> > > > >> the `-Werror` javac flag to be sure that they don't miss those
> > > warnings.
> > > > >> Given our release cadence, this means that they will have about
> six
> > > > months
> > > > >> to migrate off of any deprecated APIs without being forced to
> avoid
> > > > >> updates. And they really need to be able to adopt further minor
> > > updates
> > > > in
> > > > >> the series, since they may contain further improvements that
> > actually
> > > > >> facilitate migrating off of the deprecated APIs.
> > > > >>
> > > > >> The compiler warnings are a far more reliable mechanism to
> advertise
> > > > >> deprecation than release notes, since users' own builds will
> notify
> > > them
> > > > >> right away about only the deprecations that are relevant to them,
> > but
> > > it
> > > > >> does rely on users feeling ok to bump up minor versions in a
> timely
> > > > >> fashion. If we consistently burn them by not observing the
> stability
> > > > >> guarantees, they'll try to avoid updates instead, and the whole
> > thing
> > > > falls
> > > > >> apart.
> > > > >>
> > > > >> Thanks again,
> > > > >> -John
> > > > >>
> > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > >>> Hi Xintong,
> > > > >>>
> > > > >>> Let's compare the following cases:
> > > > >>>
> > > > >>> A. If the maintenance overhead of the deprecated API is high, and
> > we
> > > > want
> > > > >>> to remove it after two minor releases. Then there are two
> options:
> > > > >>>    A1: Demote the API in the major version bump and remove the
> code
> > > > >> with a
> > > > >>> minor version bump.
> > > > >>>    A2: Do exactly the same as 1-A, except that when removing the
> > > code,
> > > > >>> bump up the major version. So this might be a short major
> version.
> > > > >>>
> > > > >>> B. If the maintenance overhead of the deprecated API is not high,
> > > > >> therefore
> > > > >>> keeping it for a long time is affordable. There are also two
> > options:
> > > > >>>    B1: Same as A-1, demote the API in the major version bump and
> > > remove
> > > > >>> the code with a minor version bump.
> > > > >>>    B2: Keep the API for all the minor versions in the major
> > version,
> > > > and
> > > > >>> only remove the code in the next major version.
> > > > >>>
> > > > >>> For case B, do we agree that B2 is the way to go?
> > > > >>>
> > > > >>> For case A:
> > > > >>> The following stays the same for A1 and A2:
> > > > >>>  - users will lose the API after two minor leases. So the
> migration
> > > > >> period
> > > > >>> is the same for A-1 and A-2.
> > > > >>>  - A1 and A2 will have exactly the same code after the removal of
> > > > >>> deprecated API, the only difference is versioning.
> > > > >>>  - To move forward, users need to move to the next minor version
> in
> > > A1,
> > > > >> or
> > > > >>> to the next major version in A2. Because the code are the same,
> the
> > > > >> actual
> > > > >>> effort to upgrade is the same for A1 and A2.
> > > > >>>
> > > > >>> The differences between A-1 and A-2 are:
> > > > >>>  - A1 allows keeping the major version release cadence. A-2 will
> > > have a
> > > > >>> short major version release.
> > > > >>>  - A1 breaks the well understood API semantic, while A-2 does
> not.
> > > > >>>
> > > > >>> From what I see, since there is no well establish standard
> > regarding
> > > > how
> > > > >>> long a major version should be released, A short major version
> > > release
> > > > is
> > > > >>> potential and emotional. It is not ideal but does not have
> material
> > > > >>> downsides compared with A1. I did not hear anyone complaining
> about
> > > > Kafka
> > > > >>> only has two 1.x release. However, A1 actually breaks the well
> > > > understood
> > > > >>> API semantic, which has more material impact.
> > > > >>>
> > > > >>> Also, I'd imagine 90% or more of the Public APIs should fall into
> > > case
> > > > B.
> > > > >>> So, short major versions should be very occasional. I'd be very
> > > > concerned
> > > > >>> if the reason we choose A1 is simply because we cannot afford
> > > > >> maintaining a
> > > > >>> bunch of deprecated APIs until the next major version. This
> > indicates
> > > > >> that
> > > > >>> the actual problem we need to solve is to lower the maintenance
> > > > overhead
> > > > >> of
> > > > >>> deprecated APIs, so that we are comfortable to keep them longer.
> As
> > > > John
> > > > >>> and I mentioned earlier, there are ways to achieve this and we
> need
> > > to
> > > > >>> learn how to do it in Flink. Otherwise, our discussion about
> > > versioning
> > > > >>> here does not bring much value, because we will end up with a
> bunch
> > > of
> > > > >>> short-lived APIs which upset our users, no matter how we version
> > the
> > > > >>> releases.
> > > > >>>
> > > > >>> So, if there are concrete examples that you think will block us
> > from
> > > > >>> keeping API stability with affordable cost, let's take a look
> > > together
> > > > >> and
> > > > >>> see if that can be improved.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Jiangjie (Becket) Qin
> > > > >>>
> > > > >>>
> > > > >>> From what I see,
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > tonysong820@gmail.com
> > > > <ma...@gmail.com>>
> > > > >> wrote:
> > > > >>>
> > > > >>>>>
> > > > >>>>> The part I don't understand is if we are willing to have a
> > > migration
> > > > >>>>> period, and do a minor version bump to remove an API, what do
> we
> > > > >> lose to
> > > > >>>> do
> > > > >>>>> a major version bump instead, so we don't break the common
> > > versioning
> > > > >>>>> semantic?
> > > > >>>>>
> > > > >>>>
> > > > >>>> I think we are talking about the cases where a major version
> bump
> > > > >> happens
> > > > >>>> before a deprecated Public API reaches its migration period. So
> > > > >> removing
> > > > >>>> the API with another major version bump means we have two
> > > consecutive
> > > > >> major
> > > > >>>> versions in a very short time. And as previously mentioned,
> having
> > > > >> frequent
> > > > >>>> major version bumps would weaken the value of the commitment
> > "Public
> > > > >> API
> > > > >>>> stay compatible within a major version". I think users also have
> > > > >>>> expectations about how long a major version should live, which
> > > should
> > > > >> be at
> > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > >>>>
> > > > >>>> This is another option, but I think it is very likely more
> > expensive
> > > > >> than
> > > > >>>>> simply bumping the major version. And I imagine there will be
> > > > >> questions
> > > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > > > >>>>>
> > > > >>>>
> > > > >>>> Yes, it's more expensive than simply bumping the major version,
> > but
> > > > >> IMHO
> > > > >>>> it's probably cheaper than carrying the API until the next major
> > > > >> version if
> > > > >>>> we don't bump the major version very soon. And see my reply
> above
> > on
> > > > >> why
> > > > >>>> not bumping immediately.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>>
> > > > >>>> Xintong
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> becket.qin@gmail.com
> > > > <ma...@gmail.com>>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> Hi Xintong,
> > > > >>>>>
> > > > >>>>> Please see the replies below.
> > > > >>>>>
> > > > >>>>> I see your point, that users would feel surprised if they find
> > > > >> things no
> > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > However,
> > > > >> I'd
> > > > >>>>> like
> > > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > > >> problem
> > > > >>>>>> anyway. So the question is, how do we catch users' attention
> and
> > > > >> make
> > > > >>>>> sure
> > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> > Public
> > > > >> in
> > > > >>>>> 2.0.
> > > > >>>>>> There're various ways to do that, e.g., release notes,
> warnings
> > in
> > > > >>>> logs,
> > > > >>>>>> etc.
> > > > >>>>>
> > > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs in
> > > minor
> > > > >>>>> version changes. Personally speaking, I want them to be also
> > > removed
> > > > >> in
> > > > >>>>> major version changes.
> > > > >>>>>
> > > > >>>>> Empirically, I rarely see projects with complex API semantic
> work
> > > > >> well.
> > > > >>>>> Many, if not most, users don't read docs / release notes /
> > warnings
> > > > >> in
> > > > >>>>> logs. I don't think that is their fault. Typically an
> application
> > > > >>>> developer
> > > > >>>>> will have to deal with dozens of libraries maintained by
> various
> > > > >>>>> communities / groups. It is just too difficult for them to keep
> > > > >> track of
> > > > >>>>> all the specific semantics each project puts there. If you
> think
> > > > >> about
> > > > >>>> it,
> > > > >>>>> how many of Flink developers read all the release notes of
> Guava,
> > > > >> Apache
> > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version
> > of
> > > > >> them?
> > > > >>>>> One would probably try bumping the dependency version and see
> if
> > > > >> there is
> > > > >>>>> an exception and solve them case by case. And if it is a minor
> > > > >> version
> > > > >>>>> bump, one probably does not expect an exception at all. Another
> > > > >> example
> > > > >>>> is
> > > > >>>>> that in Flink we still have so many usage of deprecated methods
> > all
> > > > >> over
> > > > >>>>> the place, and I strongly doubt everyone knows when the source
> > code
> > > > >> of
> > > > >>>>> these methods are deprecated and when they should be removed.
> > > > >>>>>
> > > > >>>>> So, the most important principle of API is simple and
> intuitive.
> > > The
> > > > >>>>> versioning semantic is a simple and universally accepted API
> > > > >> stability
> > > > >>>>> standard. If we ourselves as Flink developers are relying on
> this
> > > > >> for our
> > > > >>>>> own dependencies. I don't think we can expect more from our
> > users.
> > > > >>>>>
> > > > >>>>> Another possible alternative: whenever there's a deprecated
> > Public
> > > > >> API
> > > > >>>> that
> > > > >>>>>> reaches a major version bump before the migration period, and
> we
> > > > >> also
> > > > >>>>> don't
> > > > >>>>>> want to carry it for all the next major release series, we may
> > > > >> consider
> > > > >>>>>> releasing more minor releases for the previous major version
> > after
> > > > >> the
> > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> > bump
> > > > >> to
> > > > >>>> 2.0,
> > > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> > users
> > > > >>>> another
> > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > >>>> 2-minor-release
> > > > >>>>>> migration period.
> > > > >>>>>
> > > > >>>>> This is another option, but I think it is very likely more
> > > expensive
> > > > >> than
> > > > >>>>> simply bumping the major version. And I imagine there will be
> > > > >> questions
> > > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > > > >>>>>
> > > > >>>>> I think my major point is, we should not carry APIs deprecated
> > in a
> > > > >>>>>> previous major version along all the next major version
> series.
> > > I'd
> > > > >>>> like
> > > > >>>>> to
> > > > >>>>>> try giving users more commitments, i.e. the migration period,
> as
> > > > >> long
> > > > >>>> as
> > > > >>>>> it
> > > > >>>>>> does not prevent us from making breaking changes. If it
> doesn't
> > > > >> work,
> > > > >>>> I'd
> > > > >>>>>> be in favor of not providing the migration period, but
> fallback
> > to
> > > > >> only
> > > > >>>>>> guarantee the compatibility within the major version.
> > > > >>>>>
> > > > >>>>> The part I don't understand is if we are willing to have a
> > > migration
> > > > >>>>> period, and do a minor version bump to remove an API, what do
> we
> > > > >> lose to
> > > > >>>> do
> > > > >>>>> a major version bump instead, so we don't break the common
> > > versioning
> > > > >>>>> semantic?
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>>
> > > > >>>>> Jiangjie (Becket) Qin
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > tonysong820@gmail.com
> > > > <ma...@gmail.com>>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>>>
> > > > >>>>>>> As an end user who only uses Public APIs, if I don't change
> my
> > > > >> code
> > > > >>>> at
> > > > >>>>>>> all, my expectation is the following:
> > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> fine
> > > > >> with
> > > > >>>>> all
> > > > >>>>>>> the 2.x versions.
> > > > >>>>>>> Actually I think there are some dependency version resolution
> > > > >>>> policies
> > > > >>>>>> out
> > > > >>>>>>> there which picks the highest minor version when the
> > dependencies
> > > > >>>> pull
> > > > >>>>> in
> > > > >>>>>>> multiple minor versions of the same jar, which may be broken
> if
> > > > >> we
> > > > >>>>> remove
> > > > >>>>>>> the API in minor releases.
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> I see your point, that users would feel surprised if they find
> > > > >> things
> > > > >>>> no
> > > > >>>>>> longer work when upgrading to another 2.x minor release.
> > However,
> > > > >> I'd
> > > > >>>>> like
> > > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > > >> problem
> > > > >>>>>> anyway. So the question is, how do we catch users' attention
> and
> > > > >> make
> > > > >>>>> sure
> > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> > Public
> > > > >> in
> > > > >>>>> 2.0.
> > > > >>>>>> There're various ways to do that, e.g., release notes,
> warnings
> > in
> > > > >>>> logs,
> > > > >>>>>> etc.
> > > > >>>>>>
> > > > >>>>>> Another possible alternative: whenever there's a deprecated
> > Public
> > > > >> API
> > > > >>>>> that
> > > > >>>>>> reaches a major version bump before the migration period, and
> we
> > > > >> also
> > > > >>>>> don't
> > > > >>>>>> want to carry it for all the next major release series, we may
> > > > >> consider
> > > > >>>>>> releasing more minor releases for the previous major version
> > after
> > > > >> the
> > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> > bump
> > > > >> to
> > > > >>>> 2.0,
> > > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> > users
> > > > >>>> another
> > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > > >>>> 2-minor-release
> > > > >>>>>> migration period.
> > > > >>>>>>
> > > > >>>>>> I think my major point is, we should not carry APIs deprecated
> > in
> > > a
> > > > >>>>>> previous major version along all the next major version
> series.
> > > I'd
> > > > >>>> like
> > > > >>>>> to
> > > > >>>>>> try giving users more commitments, i.e. the migration period,
> as
> > > > >> long
> > > > >>>> as
> > > > >>>>> it
> > > > >>>>>> does not prevent us from making breaking changes. If it
> doesn't
> > > > >> work,
> > > > >>>> I'd
> > > > >>>>>> be in favor of not providing the migration period, but
> fallback
> > to
> > > > >> only
> > > > >>>>>> guarantee the compatibility within the major version.
> > > > >>>>>>
> > > > >>>>>> Best,
> > > > >>>>>>
> > > > >>>>>> Xintong
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > vvcephei@apache.org
> > > > <ma...@apache.org>
> > > > >>>
> > > > >>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Becket,
> > > > >>>>>>>
> > > > >>>>>>> Thanks for the reply! I’d like to continue the conversation
> > about
> > > > >>>>>>> compatibility outside of this FLIP thread, but for now, I can
> > > > >> accept
> > > > >>>>> your
> > > > >>>>>>> decision. It’s certainly an improvement.
> > > > >>>>>>>
> > > > >>>>>>> Thanks again,
> > > > >>>>>>> John
> > > > >>>>>>>
> > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > >>>>>>>> Hi John,
> > > > >>>>>>>>
> > > > >>>>>>>> Completely agree with all you said.
> > > > >>>>>>>>
> > > > >>>>>>>> Can we consider only dropping deprecated APIs in major
> > releases
> > > > >>>>> across
> > > > >>>>>>> the
> > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving
> APIs
> > > > >> are
> > > > >>>> by
> > > > >>>>>>>>> definition less stable, but it seems like this should be
> > > > >> reflected
> > > > >>>>> in
> > > > >>>>>>> the
> > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> > them
> > > > >>>>> around
> > > > >>>>>>> for
> > > > >>>>>>>>> at least zero or one minor release, not that we can drop
> them
> > > > >> in a
> > > > >>>>>>> minor or
> > > > >>>>>>>>> patch release.
> > > > >>>>>>>>
> > > > >>>>>>>> Personally speaking, I would love to do this, for exactly
> the
> > > > >>>> reason
> > > > >>>>>> you
> > > > >>>>>>>> mentioned. However, I did not propose this due to the
> > following
> > > > >>>>>> reasons:
> > > > >>>>>>>>
> > > > >>>>>>>> 1. I am hesitating a little bit about changing the accepted
> > > > >> FLIPs
> > > > >>>> too
> > > > >>>>>>> soon.
> > > > >>>>>>>> 2. More importantly, to avoid slowing down our development.
> At
> > > > >> this
> > > > >>>>>>> point,
> > > > >>>>>>>> Flink still lacks some design / routines to support good API
> > > > >>>>>>> evolvability /
> > > > >>>>>>>> extensibility. Just like you said, it takes some time to be
> > > > >> good at
> > > > >>>>>> this.
> > > > >>>>>>>> In this case, my concern is that only removing Experimental
> /
> > > > >>>>>>>> PublicEvolving APIs in major version changes may result in
> too
> > > > >> much
> > > > >>>>>>>> overhead and dramatically slow down the development of
> Flink.
> > > > >> So, I
> > > > >>>>> was
> > > > >>>>>>>> thinking that we can start with the current status.
> Hopefully
> > > > >> after
> > > > >>>>> we
> > > > >>>>>>> are
> > > > >>>>>>>> more comfortable with the maintenance overhead of deprecated
> > > > >> APIs,
> > > > >>>> we
> > > > >>>>>> can
> > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > >> PublicEvolving
> > > > >>>>> APIs.
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks,
> > > > >>>>>>>>
> > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > >> vvcephei@apache.org <ma...@apache.org>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi Becket,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is
> really
> > > > >>>>>> important.
> > > > >>>>>>> I
> > > > >>>>>>>>> understand some people’s concerns about the additional
> burden
> > > > >> for
> > > > >>>>>>> project
> > > > >>>>>>>>> maintainers, but my personal experience with Kafka has been
> > > > >> that
> > > > >>>>> it’s
> > > > >>>>>>> very
> > > > >>>>>>>>> liveable and that it’s well worth the benefit to users. In
> > > > >> fact,
> > > > >>>>> users
> > > > >>>>>>>>> being able to confidently upgrade is also a benefit to
> > > > >>>> maintainers,
> > > > >>>>> as
> > > > >>>>>>> we
> > > > >>>>>>>>> will get fewer questions from people stuck on very old
> > > > >> versions.
> > > > >>>>>>>>>
> > > > >>>>>>>>> One question:
> > > > >>>>>>>>> Can we consider only dropping deprecated APIs in major
> > > > >> releases
> > > > >>>>> across
> > > > >>>>>>> the
> > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving
> APIs
> > > > >> are
> > > > >>>> by
> > > > >>>>>>>>> definition less stable, but it seems like this should be
> > > > >> reflected
> > > > >>>>> in
> > > > >>>>>>> the
> > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> > them
> > > > >>>>> around
> > > > >>>>>>> for
> > > > >>>>>>>>> at least zero or one minor release, not that we can drop
> them
> > > > >> in a
> > > > >>>>>>> minor or
> > > > >>>>>>>>> patch release.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The advantage of forbidding the removal of any API in minor
> > or
> > > > >>>> patch
> > > > >>>>>>>>> releases is that users will get a strong guarantee that
> they
> > > > >> can
> > > > >>>>> bump
> > > > >>>>>>> the
> > > > >>>>>>>>> minor or patch version and still be able to compile, or
> even
> > > > >> just
> > > > >>>>>>> re-link
> > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run
> > > > >> time.
> > > > >>>>> This
> > > > >>>>>>> is a
> > > > >>>>>>>>> binary guarantee: if we allow removing  even Experimental
> > APIs
> > > > >>>>> outside
> > > > >>>>>>> of
> > > > >>>>>>>>> major releases, users can no longer confidently upgrade.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of
> points:
> > > > >>>>>>>>> * I’d use the official Deprecated annotation instead of
> > > > >>>> introducing
> > > > >>>>>> our
> > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > integrated
> > > > >>>> into
> > > > >>>>>>> build
> > > > >>>>>>>>> tools and IDEs.
> > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP;
> it
> > > > >> seems
> > > > >>>>>>>>> orthogonal, and something that should probably be taken
> > > > >>>> case-by-case
> > > > >>>>>>>>> anyway.
> > > > >>>>>>>>> * Aside from deprecation and removal, there have been some
> > > > >>>>> discussions
> > > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways.
> > > > >> This is
> > > > >>>>>>> somewhat
> > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before,
> > it’ll
> > > > >>>> take
> > > > >>>>>> some
> > > > >>>>>>>>> time to become good at it. I feel like this topic should
> also
> > > > >> be
> > > > >>>>>>> orthogonal
> > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
> > > > >> simple
> > > > >>>>>> policy
> > > > >>>>>>> not
> > > > >>>>>>>>> to break existing user programs, and leave the “how” up to
> > > > >>>>>> implementers
> > > > >>>>>>> and
> > > > >>>>>>>>> reviewers.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks again,
> > > > >>>>>>>>> John
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > >>>>>>>>>> Hi All,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
> > > > >> great
> > > > >>>>> idea.
> > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > >>>> @PublicEvolving
> > > > >>>>>> --(2
> > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > > > >> flexibility
> > > > >>>>>>> without
> > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> > > > >> change
> > > > >>>>>> between
> > > > >>>>>>>>> major
> > > > >>>>>>>>>> releases. Changing annotations is acceptable and provides
> > > > >>>>> additional
> > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself
> > > > >> are
> > > > >>>> not
> > > > >>>>>>>>> changed.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I
> > > > >> want to
> > > > >>>>>> move
> > > > >>>>>>> one
> > > > >>>>>>>>>> step further and suggest introducing one more annotation
> > > > >>>> @Retired.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > > > >> downgrading
> > > > >>>>>>> @Public
> > > > >>>>>>>>> to
> > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java
> > > > >>>> standard
> > > > >>>>>>>>>> @deprecated should be used in the early stage of the
> > > > >> deprecation
> > > > >>>>>>> process
> > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does
> not
> > > > >>>> allow
> > > > >>>>>> us
> > > > >>>>>>> to
> > > > >>>>>>>>>> extend annotation, I think it would be feasible to have
> the
> > > > >> new
> > > > >>>>>>> @Retired
> > > > >>>>>>>>> to
> > > > >>>>>>>>>> help us monitor and manage the deprecation process, house
> > > > >>>>> cleaning,
> > > > >>>>>>> etc.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> @Retired:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > >> compatibility
> > > > >>>>>> before
> > > > >>>>>>>>> APIs
> > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes
> > > > >> on
> > > > >>>> the
> > > > >>>>> ML
> > > > >>>>>>> are
> > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
> > > > >> release
> > > > >>>>>>> sharply
> > > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether
> it
> > > > >> is
> > > > >>>>>>> possible).
> > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to remove
> > > > >> them.
> > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> @Retired
> > > > >> are
> > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
> > > > >> @Retired
> > > > >>>>>> could
> > > > >>>>>>>>> also
> > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > >>>> dev-friendliness,
> > > > >>>>>> but
> > > > >>>>>>> not
> > > > >>>>>>>>>> mandatory.
> > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support
> the
> > > > >>>>>>> deprecation
> > > > >>>>>>>>>> process management. Further extension is possible, since
> the
> > > > >>>>>>> annotation
> > > > >>>>>>>>> is
> > > > >>>>>>>>>> built by us.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best regards,
> > > > >>>>>>>>>> Jing
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > > >>>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hi Xintong,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies inline
> > > > >>>> below.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public API
> > > > >> is
> > > > >>>>> also a
> > > > >>>>>>>>> kind of
> > > > >>>>>>>>>>>> such change, just like removing one, which can only
> > > > >> happen
> > > > >>>> with
> > > > >>>>>>> major
> > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> Public
> > > > >>>> APIs
> > > > >>>>>>>>> anytime,
> > > > >>>>>>>>>>> but
> > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > >> reaching
> > > > >>>> the
> > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > >> would
> > > > >>>> be a
> > > > >>>>>>> weaker
> > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > >> major
> > > > >>>>>> version
> > > > >>>>>>>>> bumps,
> > > > >>>>>>>>>>>> in order to keep the commitment about the
> 2-minor-release
> > > > >>>>>> migration
> > > > >>>>>>>>>>> period.
> > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > > >> sounds
> > > > >>>>>>> against
> > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > >> necessary,
> > > > >>>>>> e.g.,
> > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> > > > >> change my
> > > > >>>>> code
> > > > >>>>>>> at
> > > > >>>>>>>>> all,
> > > > >>>>>>>>>>> my expectation is the following:
> > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I
> am
> > > > >> fine
> > > > >>>>>> with
> > > > >>>>>>> all
> > > > >>>>>>>>>>> the 2.x versions.
> > > > >>>>>>>>>>> Actually I think there are some dependency version
> > > > >> resolution
> > > > >>>>>>> policies
> > > > >>>>>>>>> out
> > > > >>>>>>>>>>> there which picks the highest minor version when the
> > > > >>>> dependencies
> > > > >>>>>>> pull
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>> multiple minor versions of the same jar, which may be
> > > > >> broken if
> > > > >>>>> we
> > > > >>>>>>>>> remove
> > > > >>>>>>>>>>> the API in minor releases.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> that
> > > > >> we
> > > > >>>>> bump
> > > > >>>>>> up
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > >> However,
> > > > >>>>> this
> > > > >>>>>>> also
> > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> will
> > > > >>>> stay
> > > > >>>>>>> stable
> > > > >>>>>>>>>>>> within the major release series, as the series can end
> > > > >>>> anytime.
> > > > >>>>>>> IMHO,
> > > > >>>>>>>>>>> short
> > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > >> users
> > > > >>>>>> happy",
> > > > >>>>>>> but
> > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > >> frequent
> > > > >>>>>> breaking
> > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > > > >> can
> > > > >>>>>> still
> > > > >>>>>>>>> have
> > > > >>>>>>>>>>>> longer major release series, while only allowing Public
> > > > >> APIs
> > > > >>>>>>>>> deprecated
> > > > >>>>>>>>>>> at
> > > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > > >> the
> > > > >>>> next
> > > > >>>>>>> major
> > > > >>>>>>>>>>>> version.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I totally agree that frequent major version bumps are not
> > > > >>>> ideal,
> > > > >>>>>> but
> > > > >>>>>>>>> here
> > > > >>>>>>>>>>> we are comparing it with a minor version bump which
> > > > >> removes a
> > > > >>>>>> Public
> > > > >>>>>>>>> API.
> > > > >>>>>>>>>>> So the context is that we have already decided to remove
> > > > >> this
> > > > >>>>>> Public
> > > > >>>>>>> API
> > > > >>>>>>>>>>> while keeping everything else backwards compatible. I
> > > > >> think a
> > > > >>>>> major
> > > > >>>>>>>>> version
> > > > >>>>>>>>>>> bump is a commonly understood signal here, compared with
> a
> > > > >>>> minor
> > > > >>>>>>> version
> > > > >>>>>>>>>>> change. From end users' perspective, for those who are
> not
> > > > >>>>>> impacted,
> > > > >>>>>>> in
> > > > >>>>>>>>>>> this case upgrading a major version is not necessarily
> more
> > > > >>>>>> involved
> > > > >>>>>>>>> than
> > > > >>>>>>>>>>> upgrading a minor version - both should be as smooth as a
> > > > >>>>>> dependency
> > > > >>>>>>>>>>> version change. For those who are impacted, they will
> lose
> > > > >> the
> > > > >>>>>> Public
> > > > >>>>>>>>> API
> > > > >>>>>>>>>>> anyways and a major version bump ensures there is no
> > > > >> surprise.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > > >>>>>>>>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Public API is a well defined common concept, and one of
> > > > >> its
> > > > >>>>>>>>>>>>> convention is that it only changes with a major version
> > > > >>>>> change.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public
> > > > >> API is
> > > > >>>>>> also a
> > > > >>>>>>>>> kind
> > > > >>>>>>>>>>>> of such change, just like removing one, which can only
> > > > >> happen
> > > > >>>>>> with
> > > > >>>>>>>>> major
> > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting
> Public
> > > > >>>> APIs
> > > > >>>>>>>>> anytime,
> > > > >>>>>>>>>>> but
> > > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > > >> reaching
> > > > >>>> the
> > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > > >> would
> > > > >>>> be a
> > > > >>>>>>> weaker
> > > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > > >> major
> > > > >>>>>> version
> > > > >>>>>>>>> bumps,
> > > > >>>>>>>>>>>> in order to keep the commitment about the
> 2-minor-release
> > > > >>>>>> migration
> > > > >>>>>>>>>>> period.
> > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > > >> sounds
> > > > >>>>>>> against
> > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > > >> necessary,
> > > > >>>>>> e.g.,
> > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> But it should be
> > > > >>>>>>>>>>>>> completely OK to bump up the major version if we really
> > > > >> want
> > > > >>>>> to
> > > > >>>>>>> get
> > > > >>>>>>>>> rid
> > > > >>>>>>>>>>> of
> > > > >>>>>>>>>>>>> a public API, right?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > > > >> that we
> > > > >>>>>> bump
> > > > >>>>>>> up
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > > >> However,
> > > > >>>>> this
> > > > >>>>>>> also
> > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs
> will
> > > > >>>> stay
> > > > >>>>>>> stable
> > > > >>>>>>>>>>>> within the major release series, as the series can end
> > > > >>>> anytime.
> > > > >>>>>>> IMHO,
> > > > >>>>>>>>>>> short
> > > > >>>>>>>>>>>> major release series are not something "make the end
> > > > >> users
> > > > >>>>>> happy",
> > > > >>>>>>> but
> > > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > > >> frequent
> > > > >>>>>> breaking
> > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > > > >> can
> > > > >>>>>> still
> > > > >>>>>>>>> have
> > > > >>>>>>>>>>>> longer major release series, while only allowing Public
> > > > >> APIs
> > > > >>>>>>>>> deprecated
> > > > >>>>>>>>>>> at
> > > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > > >> the
> > > > >>>> next
> > > > >>>>>>> major
> > > > >>>>>>>>>>>> version.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > > >> (1-2
> > > > >>>>> years)
> > > > >>>>>>> to
> > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put
> it
> > > > >>>> again
> > > > >>>>>> to
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation
> on
> > > > >>>> when
> > > > >>>>>>>>> breaking
> > > > >>>>>>>>>>>>> changes can be made.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
> > > > >> for new
> > > > >>>>>> major
> > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > > > >> needed.
> > > > >>>> That
> > > > >>>>>>> makes
> > > > >>>>>>>>> 1-2
> > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving
> > > > >> us
> > > > >>>> more
> > > > >>>>>>> time
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Xintong
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
> > > > >>>>> versions
> > > > >>>>>>> for
> > > > >>>>>>>>>>>>> previous
> > > > >>>>>>>>>>>>>> major versions.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> > > > >> viable
> > > > >>>> that
> > > > >>>>>> is
> > > > >>>>>>> in
> > > > >>>>>>>>>>>>> practice.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > > > >> support
> > > > >>>> for
> > > > >>>>>> Java
> > > > >>>>>>>>> 8/11,
> > > > >>>>>>>>>>>>> which may very well be a problem for our current users.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
> > > > >> about
> > > > >>>>>> Public
> > > > >>>>>>>>> API
> > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
> > > > >> in
> > > > >>>> the
> > > > >>>>>>> NOTES
> > > > >>>>>>>>> part
> > > > >>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
> > > > >> it.
> > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
> > > > >> are on
> > > > >>>>> the
> > > > >>>>>>> same
> > > > >>>>>>>>>>>>> page on that note.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > >>>>>>>>>>>>>> But it should be
> > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > > > >> really
> > > > >>>> want
> > > > >>>>>> to
> > > > >>>>>>> get
> > > > >>>>>>>>>>> rid
> > > > >>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>> a public API, right?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get us
> > > > >> to
> > > > >>>>> 2.0.
> > > > >>>>>> ;)
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> There's a separate discussion to be had on the cadence
> > > > >> of
> > > > >>>>> major
> > > > >>>>>>>>> releases
> > > > >>>>>>>>>>>>> going forward, and there seem to be different opinions
> > > > >> on
> > > > >>>>> that.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases
> between
> > > > >>>> major
> > > > >>>>>>> ones,
> > > > >>>>>>>>>>>>> that for us means that users have to potentially deal
> > > > >> with
> > > > >>>>>>> breaking
> > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > > >> (1-2
> > > > >>>>>> years)
> > > > >>>>>>> to
> > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put
> it
> > > > >>>> again
> > > > >>>>>> to
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation
> on
> > > > >>>> when
> > > > >>>>>>>>> breaking
> > > > >>>>>>>>>>>>> changes can be made.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
> > > > >> Not
> > > > >>>>> sure
> > > > >>>>>> I
> > > > >>>>>>>>> fully
> > > > >>>>>>>>>>>>>> understand the concern for behavioral change here.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> This could be a lot of things. It can be performance in
> > > > >>>>> certain
> > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
> > > > >> relied
> > > > >>>>> upon
> > > > >>>>>>>>>>>>> (
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > ),
> > > > a semantic change to some
> > > > >> API.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> For a concrete example, consider the job submission. A
> > > > >> few
> > > > >>>>>>> releases
> > > > >>>>>>>>> back
> > > > >>>>>>>>>>>>> we made changes such that the initialization of the job
> > > > >>>> master
> > > > >>>>>>>>> happens
> > > > >>>>>>>>>>>>> asynchronously.
> > > > >>>>>>>>>>>>> This meant the job submission call returns sooner, and
> > > > >> the
> > > > >>>> job
> > > > >>>>>>> state
> > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but the
> > > > >>>>> observed
> > > > >>>>>>>>> behavior
> > > > >>>>>>>>>>>>> may be different.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Metrics are another example; I believe over time we
> > > > >> changed
> > > > >>>>> what
> > > > >>>>>>> some
> > > > >>>>>>>>>>>>> metrics returned a few times.
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jingsong Li <ji...@gmail.com>.
Thanks Becket and all for your discussion.

> 1. We say this FLIP is enforced starting release 2.0. For current 1.x APIs,
we provide a migration period with best effort, while allowing exceptions
for immediate removal in 2.0. That means we will still try with best effort
to get the ProcessFuncion API ready and deprecate the DataStream API in
1.x, but will also be allowed to remove DataStream API in 2.0 if it's not
deprecated 2 minor releases before the major version bump.

> 2. We strictly follow the process in this FLIP, and will quickly bump the
major version from 2.x to 3.0 once the migration period for DataStream API
is reached.

Sorry, I didn't read the previous detailed discussion because the
discussion list was so long.

I don't really like either of these options.

Considering that DataStream is such an important API, can we offer a third
option:

3. Maintain the DataStream API throughout 2.X and remove it until 3.x. But
there's no need to assume that 2.X is a short version, it's still a normal
major version.

Best,
Jingsong

Becket Qin <be...@gmail.com>于2023年6月22日 周四16:02写道:

> Thanks much for the input, John, Stefan and Jing.
>
> I think Xingtong has well summarized the pros and cons of the two options.
> Let's collect a few more opinions here and we can move forward with the one
> more people prefer.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi all,
> >
> > Thanks Xingtong for the summary. If I could only choose one of the given
> > two options, I would go with option 1. I understood that option 2 worked
> > great with Kafka. But the bridge release will still confuse users and my
> > gut feeling is that many users will skip 2.0 and be waiting for 3.0 or
> even
> > 3.x. And since fewer users will use Flink 2.x, the development focus will
> > be on Flink 3.0 with the fact that the current Flink release is 1.17 and
> we
> > are preparing 2.0 release. That is weird for me.
> >
> > THB, I would not name the change from @Public to @Retired as a demotion.
> > The purpose of @Retire is to extend the API lifecycle with one more
> stage,
> > like in the real world, people born, studied, graduated, worked, and
> > retired. Afaiu from the previous discussion, there are two rules we'd
> like
> > to follow simultaneously:
> >
> > 1. Public APIs can only be changed between major releases.
> > 2. A smooth migration phase should be offered to users, i.e. at least 2
> > minor releases after APIs are marked as @deprecated. There should be new
> > APIs as the replacement.
> >
> > Agree, those rules are good to improve the user friendliness. Issues we
> > discussed are rising because we want to fulfill both of them. If we take
> > care of deprecation very seriously, APIs can be marked as @Deprecated,
> only
> > when the new APIs as the replacement provide all functionalities the
> > deprecated APIs have. In an ideal case without critical bugs that might
> > stop users adopting the new APIs. Otherwise the expected "replacement"
> will
> > not happen. Users will still stick to the deprecated APIs, because the
> new
> > APIs can not be used. For big features, it will need at least 4 minor
> > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> >
> > - 1st minor release to build the new APIs as the replacement and waiting
> > for feedback. It might be difficult to mark the old API as deprecated in
> > this release, because we are not sure if the new APIs could cover 100%
> > functionalities.
> > -  In the lucky case,  mark all old APIs as deprecated in the 2nd minor
> > release. (I would even suggest having the new APIs released at least for
> > two minor releases before marking it as deprecated to make sure they can
> > really replace the old APIs, in case we care more about smooth migration)
> > - 3rd minor release for the migration period
> > -  In another lucky case, the 4th release is a major release, the
> > deprecated APIs could be removed.
> >
> > The above described scenario works only in an ideal case. In reality, it
> > might take longer to get the new APIs ready and mark the old API
> > deprecated. Furthermore, if the 4th release is not a major release, we
> will
> > have to maintain both APIs for many further minor releases. The question
> is
> > how to know the next major release in advance, especially 4 minor
> releases'
> > period, i.e. more than 2 years in advance? Given that Flink contains many
> > modules, it is difficult to ask devs to create a 2-3 years deprecation
> plan
> > for each case. In case we want to build major releases at a fast pace,
> > let's say every two years, it means devs must plan any API deprecation
> > right after each major release. Afaiac, it is quite difficult.
> >
> > The major issue is, afaiu, if we follow rule 2, we have to keep all
> @Public
> > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0.
> Then
> > we have to follow rule 1 to keep it unchanged until we have 3.0. That is
> > why @Retired is useful to give devs more flexibility and still fulfill
> both
> > rules. Let's check it with examples:
> >
> > - we have @Public DataStream API in 1.18. It will not be marked
> > as @Deprecated, because the new APIs as the replacement are not ready.
> > - we keep the DataStream API itself unchanged in 2.0, but change the
> > annotation from @Public to @Retire. New APIs will be introduced too. In
> > this case, Rule 1 is ok, since public API is allowed to change between
> > major releases. Only changing annotation is the minimal change we could
> do
> > and it does not break rule 1. Rule 2 is ok too, since the DataStream APIs
> > work exactly the same.  Attention: the change of @Public -> @Retired can
> > only be done between major releases, because @Public APIs can only be
> > changed between major releases.
> > - in 2.1, DataStream API will be marked as deprecated.
> > - in 2.2, DataStream will be kept for the migration period.
> > - in 2.3, DataStream will be removed.
> >
> > Becket mentioned previously (please correct me if I didn't understand it
> > correctly) that users might not check the changes of annotation and the
> > upgrade process might not be smooth. I was wondering, if users don't pay
> > attention to that, they will also not pay attention to @deprecated. The
> > migration will not be smooth too even when we stick to the Java
> > standard @deprecated. Let's go throw it with examples:
> >
> > Scenario 1: with @Retired
> >
> > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be done
> > for users who still want to use DataStream. The migration is smooth. But
> > they should be aware of the annotation changes, i.e. @Public -> @Retired
> > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since it
> is
> > a major release upgrade, breaking changes are expected.
> > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed. Since
> APIs
> > are marked as @Retired, breaking changes are expected too.
> >
> > In addition to fulfilling both rules, there are two more benefits of this
> > option:
> > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > 2. no bridge release is required. DataStream API could be deprecated in
> 2.x
> > without breaking any of the two rules. @Retired provides enough
> flexibility
> > for future deprecations and upgrades.
> >
> > The only issue I can find is that users might be surprised in case 3, if
> > they ignore the previous @Public -> @Retired change. But for such users,
> > they will ignore @deprecated annotation too and will have issues with
> > pure @deprecated
> > annotation too:
> >
> > Scenario 2: with @deprecated
> >
> > The example will be slightly different from scenario 1, because @Public
> > APIs can only be removed between major releases. As an example,
> DataStream
> > API will be marked as deprecated in 1.19, kept in 1.20, and then removed
> in
> > 2.0
> >
> > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as
> > deprecated are removed between major releases. Rule 1 is ok. The 2 minor
> > releases period defined in rule 2 sounds like a smooth migration plan,
> but
> > for users in this case, it is still not smooth.
> > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with
> > deprecated APIs. But for those users who ignore annotation changes, will
> > have the same issue mentioned in case 1.
> >
> > I understood that, compared to scenario 1, users, who ignore @Public
> > -> @Retired
> > changes, will not face any breaking changes, after they migrated to 2.0.
> > But the cost of it is bridge releases. There might be many bridge
> releases
> > in the future. As I mentioned previously, it is not easy to find the
> right
> > timing to kick off the deprecation process of APIs before the next
> expected
> > major release and 2-3 years in advance.
> >
> > Scenario 3: @PublicEvolving deprecation for users who ignore @deprecated
> > annotation changes
> >
> > Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21
> >
> > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth
> > migration.
> > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > since @deprecated is ignored
> >
> > In summary, for users who care about the annotation change, they will get
> > more benefits with @Retired. For users who ignore annotation changes,
> they
> > will be always facing non-smooth upgrades even using pure standard
> > @deprecated
> > annotation/process.
> >
> > For the case that IDEs support @deprecated, that is true. There are no
> such
> > supports for homebuilt annotation. But the intention is to let users be
> > aware of it. There will be other ways to do it for users who care about
> > annotation changes. For users who do not care, there should be no
> > difference between with tool support or without. Using @Retired with
> > @deprecation
> > together is at least not worse than only using @deprecation alone. And
> > there are additional benefits with @Retired.
> >
> > Just my 2 cents and looking forward to your feedback, especially
> different
> > opinions that will help me understand the issue better. Thanks!
> >
> >
> > Best regards,
> > Jing
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > <sr...@confluent.io.invalid> wrote:
> >
> > > Hi Xintong,
> > >
> > > Thanks for the summary, most of the points that you agreed upon also
> make
> > > sense to me.
> > >
> > > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> > APIs
> > > >   from Public to PublicEvolving / Experimental / Retired in any
> version
> > > bump,
> > > >   are not good practices.
> > >
> > > I hope we can can go beyond calling it “not a good practice" and reach
> > > consensus that we will not demote or remove public APIs in minor
> releases
> > > and (technically) enforce this rule for all contributions. I think
> it’s a
> > > reasonable expectation for stability from a project as mature as Flink.
> > >
> > > > I'm personally in favor of option 1. As the migration-period rule is
> > > newly
> > > > proposed, I think it's fair to make exceptions for cases where we
> > already
> > > > missed the best chance for planning the deprecation. Moreover, I do
> > > believe
> > > > having a quick major version bump does confuse users. Yes, we can
> > explain
> > > > to users that bumping from 2.x to 3.0 does not cost anything other
> than
> > > the
> > > > removal of an deprecated API. But having to explain this itself is an
> > > > indicator that it might be confusing for users.
> > > >
> > > >
> > > > Becket, on the other hand, prefers option 2. From my understanding,
> his
> > > > major point is that a quick major version bump causes barely any
> actual
> > > > lose on users, while in option 1 not providing the migration period
> or
> > > > providing a shorter on is an actual lose on users. (@Becket, please
> > > correct
> > > > me if I'm mistaken.)
> > > >
> > >
> > > I would be open to both options and maybe postpone this decision until
> we
> > > have the complete roadmap for Flink 2.0. We would have a better
> overview
> > > about the technical consequences, e.g. how hard it would be to offer
> and
> > > maintain both APIs side by side, how well we get both APIs separated,
> and
> > > whether or not we will be able to use the full potential of our
> breaking
> > > changes before the DataStream API is completely removed. One example
> for
> > > the last point I can think of would be moving from lazy to eager state
> > > declaration. If we believe that it is reasonable to support all planned
> > > changes while maintaining both APIs, I'm leaning towards option 2.
> > >
> > > Best,
> > > Stefan
> > >
> > >
> > > > On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com>
> wrote:
> > > >
> > > > Becket and I had an offline voice call earlier today. We have reached
> > > > consensus on some of the arguments, but still hold different opinions
> > on
> > > > some others. I'd like to post the outcome here for transparency.
> > > >
> > > > We both agree that:
> > > >
> > > >   1. Providing a migration period would be beneficial for our users,
> > and
> > > >   we should do that
> > > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> > APIs
> > > >   from Public to PublicEvolving / Experimental / Retired in any
> version
> > > bump,
> > > >   are not good practices.
> > > >   3. Ideally, with this FLIP, developers should be more careful and
> > plan
> > > >   API changes ahead. That means:
> > > >      1. Be more careful with designing APIs and promoting them to
> > Public,
> > > >      so that they won't be changed / removed very soon, and also the
> > > >      maintenance overhead for keeping them after deprecation should
> be
> > > >      affordable. (I believe this also aligns with John's opinion.)
> > > >      2. Plan the deprecation / removal of old APIs that we don't want
> > to
> > > >      carry to the next major version earlier, so that they'll have
> the
> > > >      2-minor-release migration period before the next major version
> > bump.
> > > >   4. A practical situation is that, we are not ready for deprecating
> > the
> > > >   DataStream API, nor afford to carry it for another couple of years.
> > > >      1. We cannot deprecate it now, because the new ProcessFunction
> API
> > > >      (the planned replacement for DataStream API) is not yet ready.
> We
> > > haven't
> > > >      planned the ProcessFunction API earlier because the original
> > > > plan was to do
> > > >      in-place changes directly on the DataStream API in release 2.0,
> > > until the
> > > >      smooth migration period is proposed. Before this, the only
> > > guarantees we
> > > >      provide for Public APIs is that they'll stay compatible until
> the
> > > next
> > > >      major version, which led to the understanding the in-place
> changes
> > > on
> > > >      Public APIs should be fine.
> > > >      2. We also cannot afford carrying the DataStream API for another
> > > >      couple of years. One of the reasons that we want to refactor /
> > > replace
> > > >      DataStream API is that it exposes and depends on too many
> > > > Flink's internal
> > > >      runtime implementations, which has already limited / blocked
> plans
> > > for
> > > >      Flink's internal improvements for many times. Also due to the
> wide
> > > and
> > > >      transitive dependencies on the internals, maintaining two sets
> of
> > > lower
> > > >      APIs itself is expensive and would be better to keep it short.
> > This
> > > is
> > > >      admittedly a historical design flaw which should be avoided in
> > > future.
> > > >
> > > > What we haven't reached consensus is about how to deal with the
> current
> > > > situation around the DataStream API. There are two options.
> > > >
> > > >
> > > >   1. We say this FLIP is enforced starting release 2.0. For current
> 1.x
> > > >   APIs, we provide a migration period with best effort, while
> allowing
> > > >   exceptions for immediate removal in 2.0. That means we will still
> try
> > > with
> > > >   best effort to get the ProcessFuncion API ready and deprecate the
> > > >   DataStream API in 1.x, but will also be allowed to remove
> DataStream
> > > API in
> > > >   2.0 if it's not deprecated 2 minor releases before the major
> version
> > > bump.
> > > >   2. We strictly follow the process in this FLIP, and will quickly
> bump
> > > >   the major version from 2.x to 3.0 once the migration period for
> > > DataStream
> > > >   API is reached.
> > > >
> > > > I'm personally in favor of option 1. As the migration-period rule is
> > > newly
> > > > proposed, I think it's fair to make exceptions for cases where we
> > already
> > > > missed the best chance for planning the deprecation. Moreover, I do
> > > believe
> > > > having a quick major version bump does confuse users. Yes, we can
> > explain
> > > > to users that bumping from 2.x to 3.0 does not cost anything other
> than
> > > the
> > > > removal of an deprecated API. But having to explain this itself is an
> > > > indicator that it might be confusing for users.
> > > >
> > > >
> > > > Becket, on the other hand, prefers option 2. From my understanding,
> his
> > > > major point is that a quick major version bump causes barely any
> actual
> > > > lose on users, while in option 1 not providing the migration period
> or
> > > > providing a shorter on is an actual lose on users. (@Becket, please
> > > correct
> > > > me if I'm mistaken.)
> > > >
> > > >
> > > > And we'd like to hear more feedback from the community.
> > > >
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcephei@apache.org
> > > <ma...@apache.org>> wrote:
> > > >
> > > >> Hi Becket and Xintong,
> > > >>
> > > >> I hope you don't mind if I chime in a little.
> > > >>
> > > >> Once an API is marked Public, we're committing to support it until
> > it's
> > > >> deprecated, and once it's deprecated, to leave it in place for at
> > least
> > > two
> > > >> minor releases and then only remove it in the following major
> release.
> > > >>
> > > >> As a user, I would be dismayed to discover that the project
> > maintainers
> > > >> demoted a Public API to PublicEvolving or Experimental in order to
> > > violate
> > > >> the spirit of the deprecation period for Public APIs. Likewise, I'd
> > > view a
> > > >> rapid sequence of minor releases for the purpose of dropping
> > deprecated
> > > >> APIs in the next major release as another violation of the spirit of
> > our
> > > >> guarantees. I'm reminded of this xkcd:
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > >>
> > > >> I'm glad we're gaming out these situations before we institute this
> > > FLIP,
> > > >> and I hope we all converge on a clear understanding of what we're
> > > >> guaranteeing. From where I'm sitting, the essence of this FLIP is
> > > >> specifically to prevent us from doing whatever we want and instead
> to
> > > >> commit to providing a suitable notice period to users who rely on
> > Public
> > > >> APIs.
> > > >>
> > > >> It honestly shouldn't matter what the maintenance overhead is (or,
> > > rather,
> > > >> evaluating the maintenance overhead should be a consideration before
> > we
> > > >> promote an API to Public to begin with). My experience in other
> > > projects is
> > > >> that regretting a feature so much you want to drop it asap is
> > extremely
> > > >> rare, and that living with the pain for a little while until the
> > > >> deprecation period expires will probably make us better engineers.
> > > >>
> > > >> With that all said, one thing that is not a violation is to propose
> a
> > > new
> > > >> major release in order to drop especially burdensome deprecated APIs
> > > that
> > > >> have already enjoyed their full deprecation period. Those really bad
> > > "case
> > > >> A" features will be very rare.
> > > >>
> > > >> Looking over the FLIP again, I see a reference to the release
> > > >> documentation, but not to the build tooling support for the
> Deprecated
> > > >> annotation, so it might help to share this scenario: The way this
> will
> > > all
> > > >> come together in practice is that, by giving good stability
> > guarantees,
> > > we
> > > >> will encourage our users to bump minor versions when they are
> > released,
> > > >> which means that they will see compiler warnings immediately if any
> of
> > > the
> > > >> APIs they've used are deprecated. In fact, I'd recommend for them to
> > use
> > > >> the `-Werror` javac flag to be sure that they don't miss those
> > warnings.
> > > >> Given our release cadence, this means that they will have about six
> > > months
> > > >> to migrate off of any deprecated APIs without being forced to avoid
> > > >> updates. And they really need to be able to adopt further minor
> > updates
> > > in
> > > >> the series, since they may contain further improvements that
> actually
> > > >> facilitate migrating off of the deprecated APIs.
> > > >>
> > > >> The compiler warnings are a far more reliable mechanism to advertise
> > > >> deprecation than release notes, since users' own builds will notify
> > them
> > > >> right away about only the deprecations that are relevant to them,
> but
> > it
> > > >> does rely on users feeling ok to bump up minor versions in a timely
> > > >> fashion. If we consistently burn them by not observing the stability
> > > >> guarantees, they'll try to avoid updates instead, and the whole
> thing
> > > falls
> > > >> apart.
> > > >>
> > > >> Thanks again,
> > > >> -John
> > > >>
> > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > >>> Hi Xintong,
> > > >>>
> > > >>> Let's compare the following cases:
> > > >>>
> > > >>> A. If the maintenance overhead of the deprecated API is high, and
> we
> > > want
> > > >>> to remove it after two minor releases. Then there are two options:
> > > >>>    A1: Demote the API in the major version bump and remove the code
> > > >> with a
> > > >>> minor version bump.
> > > >>>    A2: Do exactly the same as 1-A, except that when removing the
> > code,
> > > >>> bump up the major version. So this might be a short major version.
> > > >>>
> > > >>> B. If the maintenance overhead of the deprecated API is not high,
> > > >> therefore
> > > >>> keeping it for a long time is affordable. There are also two
> options:
> > > >>>    B1: Same as A-1, demote the API in the major version bump and
> > remove
> > > >>> the code with a minor version bump.
> > > >>>    B2: Keep the API for all the minor versions in the major
> version,
> > > and
> > > >>> only remove the code in the next major version.
> > > >>>
> > > >>> For case B, do we agree that B2 is the way to go?
> > > >>>
> > > >>> For case A:
> > > >>> The following stays the same for A1 and A2:
> > > >>>  - users will lose the API after two minor leases. So the migration
> > > >> period
> > > >>> is the same for A-1 and A-2.
> > > >>>  - A1 and A2 will have exactly the same code after the removal of
> > > >>> deprecated API, the only difference is versioning.
> > > >>>  - To move forward, users need to move to the next minor version in
> > A1,
> > > >> or
> > > >>> to the next major version in A2. Because the code are the same, the
> > > >> actual
> > > >>> effort to upgrade is the same for A1 and A2.
> > > >>>
> > > >>> The differences between A-1 and A-2 are:
> > > >>>  - A1 allows keeping the major version release cadence. A-2 will
> > have a
> > > >>> short major version release.
> > > >>>  - A1 breaks the well understood API semantic, while A-2 does not.
> > > >>>
> > > >>> From what I see, since there is no well establish standard
> regarding
> > > how
> > > >>> long a major version should be released, A short major version
> > release
> > > is
> > > >>> potential and emotional. It is not ideal but does not have material
> > > >>> downsides compared with A1. I did not hear anyone complaining about
> > > Kafka
> > > >>> only has two 1.x release. However, A1 actually breaks the well
> > > understood
> > > >>> API semantic, which has more material impact.
> > > >>>
> > > >>> Also, I'd imagine 90% or more of the Public APIs should fall into
> > case
> > > B.
> > > >>> So, short major versions should be very occasional. I'd be very
> > > concerned
> > > >>> if the reason we choose A1 is simply because we cannot afford
> > > >> maintaining a
> > > >>> bunch of deprecated APIs until the next major version. This
> indicates
> > > >> that
> > > >>> the actual problem we need to solve is to lower the maintenance
> > > overhead
> > > >> of
> > > >>> deprecated APIs, so that we are comfortable to keep them longer. As
> > > John
> > > >>> and I mentioned earlier, there are ways to achieve this and we need
> > to
> > > >>> learn how to do it in Flink. Otherwise, our discussion about
> > versioning
> > > >>> here does not bring much value, because we will end up with a bunch
> > of
> > > >>> short-lived APIs which upset our users, no matter how we version
> the
> > > >>> releases.
> > > >>>
> > > >>> So, if there are concrete examples that you think will block us
> from
> > > >>> keeping API stability with affordable cost, let's take a look
> > together
> > > >> and
> > > >>> see if that can be improved.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jiangjie (Becket) Qin
> > > >>>
> > > >>>
> > > >>> From what I see,
> > > >>>
> > > >>>
> > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> tonysong820@gmail.com
> > > <ma...@gmail.com>>
> > > >> wrote:
> > > >>>
> > > >>>>>
> > > >>>>> The part I don't understand is if we are willing to have a
> > migration
> > > >>>>> period, and do a minor version bump to remove an API, what do we
> > > >> lose to
> > > >>>> do
> > > >>>>> a major version bump instead, so we don't break the common
> > versioning
> > > >>>>> semantic?
> > > >>>>>
> > > >>>>
> > > >>>> I think we are talking about the cases where a major version bump
> > > >> happens
> > > >>>> before a deprecated Public API reaches its migration period. So
> > > >> removing
> > > >>>> the API with another major version bump means we have two
> > consecutive
> > > >> major
> > > >>>> versions in a very short time. And as previously mentioned, having
> > > >> frequent
> > > >>>> major version bumps would weaken the value of the commitment
> "Public
> > > >> API
> > > >>>> stay compatible within a major version". I think users also have
> > > >>>> expectations about how long a major version should live, which
> > should
> > > >> be at
> > > >>>> least a couple of years rather than 1-2 minor releases.
> > > >>>>
> > > >>>> This is another option, but I think it is very likely more
> expensive
> > > >> than
> > > >>>>> simply bumping the major version. And I imagine there will be
> > > >> questions
> > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > > >>>>>
> > > >>>>
> > > >>>> Yes, it's more expensive than simply bumping the major version,
> but
> > > >> IMHO
> > > >>>> it's probably cheaper than carrying the API until the next major
> > > >> version if
> > > >>>> we don't bump the major version very soon. And see my reply above
> on
> > > >> why
> > > >>>> not bumping immediately.
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>> Xintong
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket.qin@gmail.com
> > > <ma...@gmail.com>>
> > > >> wrote:
> > > >>>>
> > > >>>>> Hi Xintong,
> > > >>>>>
> > > >>>>> Please see the replies below.
> > > >>>>>
> > > >>>>> I see your point, that users would feel surprised if they find
> > > >> things no
> > > >>>>>> longer work when upgrading to another 2.x minor release.
> However,
> > > >> I'd
> > > >>>>> like
> > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > >> problem
> > > >>>>>> anyway. So the question is, how do we catch users' attention and
> > > >> make
> > > >>>>> sure
> > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> Public
> > > >> in
> > > >>>>> 2.0.
> > > >>>>>> There're various ways to do that, e.g., release notes, warnings
> in
> > > >>>> logs,
> > > >>>>>> etc.
> > > >>>>>
> > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs in
> > minor
> > > >>>>> version changes. Personally speaking, I want them to be also
> > removed
> > > >> in
> > > >>>>> major version changes.
> > > >>>>>
> > > >>>>> Empirically, I rarely see projects with complex API semantic work
> > > >> well.
> > > >>>>> Many, if not most, users don't read docs / release notes /
> warnings
> > > >> in
> > > >>>>> logs. I don't think that is their fault. Typically an application
> > > >>>> developer
> > > >>>>> will have to deal with dozens of libraries maintained by various
> > > >>>>> communities / groups. It is just too difficult for them to keep
> > > >> track of
> > > >>>>> all the specific semantics each project puts there. If you think
> > > >> about
> > > >>>> it,
> > > >>>>> how many of Flink developers read all the release notes of Guava,
> > > >> Apache
> > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version
> of
> > > >> them?
> > > >>>>> One would probably try bumping the dependency version and see if
> > > >> there is
> > > >>>>> an exception and solve them case by case. And if it is a minor
> > > >> version
> > > >>>>> bump, one probably does not expect an exception at all. Another
> > > >> example
> > > >>>> is
> > > >>>>> that in Flink we still have so many usage of deprecated methods
> all
> > > >> over
> > > >>>>> the place, and I strongly doubt everyone knows when the source
> code
> > > >> of
> > > >>>>> these methods are deprecated and when they should be removed.
> > > >>>>>
> > > >>>>> So, the most important principle of API is simple and intuitive.
> > The
> > > >>>>> versioning semantic is a simple and universally accepted API
> > > >> stability
> > > >>>>> standard. If we ourselves as Flink developers are relying on this
> > > >> for our
> > > >>>>> own dependencies. I don't think we can expect more from our
> users.
> > > >>>>>
> > > >>>>> Another possible alternative: whenever there's a deprecated
> Public
> > > >> API
> > > >>>> that
> > > >>>>>> reaches a major version bump before the migration period, and we
> > > >> also
> > > >>>>> don't
> > > >>>>>> want to carry it for all the next major release series, we may
> > > >> consider
> > > >>>>>> releasing more minor releases for the previous major version
> after
> > > >> the
> > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> bump
> > > >> to
> > > >>>> 2.0,
> > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> users
> > > >>>> another
> > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > >>>> 2-minor-release
> > > >>>>>> migration period.
> > > >>>>>
> > > >>>>> This is another option, but I think it is very likely more
> > expensive
> > > >> than
> > > >>>>> simply bumping the major version. And I imagine there will be
> > > >> questions
> > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > > >>>>>
> > > >>>>> I think my major point is, we should not carry APIs deprecated
> in a
> > > >>>>>> previous major version along all the next major version series.
> > I'd
> > > >>>> like
> > > >>>>> to
> > > >>>>>> try giving users more commitments, i.e. the migration period, as
> > > >> long
> > > >>>> as
> > > >>>>> it
> > > >>>>>> does not prevent us from making breaking changes. If it doesn't
> > > >> work,
> > > >>>> I'd
> > > >>>>>> be in favor of not providing the migration period, but fallback
> to
> > > >> only
> > > >>>>>> guarantee the compatibility within the major version.
> > > >>>>>
> > > >>>>> The part I don't understand is if we are willing to have a
> > migration
> > > >>>>> period, and do a minor version bump to remove an API, what do we
> > > >> lose to
> > > >>>> do
> > > >>>>> a major version bump instead, so we don't break the common
> > versioning
> > > >>>>> semantic?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>>
> > > >>>>> Jiangjie (Becket) Qin
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > tonysong820@gmail.com
> > > <ma...@gmail.com>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>>>
> > > >>>>>>> As an end user who only uses Public APIs, if I don't change my
> > > >> code
> > > >>>> at
> > > >>>>>>> all, my expectation is the following:
> > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > > >> with
> > > >>>>> all
> > > >>>>>>> the 2.x versions.
> > > >>>>>>> Actually I think there are some dependency version resolution
> > > >>>> policies
> > > >>>>>> out
> > > >>>>>>> there which picks the highest minor version when the
> dependencies
> > > >>>> pull
> > > >>>>> in
> > > >>>>>>> multiple minor versions of the same jar, which may be broken if
> > > >> we
> > > >>>>> remove
> > > >>>>>>> the API in minor releases.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> I see your point, that users would feel surprised if they find
> > > >> things
> > > >>>> no
> > > >>>>>> longer work when upgrading to another 2.x minor release.
> However,
> > > >> I'd
> > > >>>>> like
> > > >>>>>> to point out that PublicEvolving APIs would have the similar
> > > >> problem
> > > >>>>>> anyway. So the question is, how do we catch users' attention and
> > > >> make
> > > >>>>> sure
> > > >>>>>> they are aware that the Public APIs in 1.x may no longer be
> Public
> > > >> in
> > > >>>>> 2.0.
> > > >>>>>> There're various ways to do that, e.g., release notes, warnings
> in
> > > >>>> logs,
> > > >>>>>> etc.
> > > >>>>>>
> > > >>>>>> Another possible alternative: whenever there's a deprecated
> Public
> > > >> API
> > > >>>>> that
> > > >>>>>> reaches a major version bump before the migration period, and we
> > > >> also
> > > >>>>> don't
> > > >>>>>> want to carry it for all the next major release series, we may
> > > >> consider
> > > >>>>>> releasing more minor releases for the previous major version
> after
> > > >> the
> > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we
> bump
> > > >> to
> > > >>>> 2.0,
> > > >>>>>> we can release one more 1.20 after 2.0. That should provide
> users
> > > >>>> another
> > > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > > >>>> 2-minor-release
> > > >>>>>> migration period.
> > > >>>>>>
> > > >>>>>> I think my major point is, we should not carry APIs deprecated
> in
> > a
> > > >>>>>> previous major version along all the next major version series.
> > I'd
> > > >>>> like
> > > >>>>> to
> > > >>>>>> try giving users more commitments, i.e. the migration period, as
> > > >> long
> > > >>>> as
> > > >>>>> it
> > > >>>>>> does not prevent us from making breaking changes. If it doesn't
> > > >> work,
> > > >>>> I'd
> > > >>>>>> be in favor of not providing the migration period, but fallback
> to
> > > >> only
> > > >>>>>> guarantee the compatibility within the major version.
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>>
> > > >>>>>> Xintong
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > vvcephei@apache.org
> > > <ma...@apache.org>
> > > >>>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Becket,
> > > >>>>>>>
> > > >>>>>>> Thanks for the reply! I’d like to continue the conversation
> about
> > > >>>>>>> compatibility outside of this FLIP thread, but for now, I can
> > > >> accept
> > > >>>>> your
> > > >>>>>>> decision. It’s certainly an improvement.
> > > >>>>>>>
> > > >>>>>>> Thanks again,
> > > >>>>>>> John
> > > >>>>>>>
> > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > >>>>>>>> Hi John,
> > > >>>>>>>>
> > > >>>>>>>> Completely agree with all you said.
> > > >>>>>>>>
> > > >>>>>>>> Can we consider only dropping deprecated APIs in major
> releases
> > > >>>>> across
> > > >>>>>>> the
> > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> > > >> are
> > > >>>> by
> > > >>>>>>>>> definition less stable, but it seems like this should be
> > > >> reflected
> > > >>>>> in
> > > >>>>>>> the
> > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> them
> > > >>>>> around
> > > >>>>>>> for
> > > >>>>>>>>> at least zero or one minor release, not that we can drop them
> > > >> in a
> > > >>>>>>> minor or
> > > >>>>>>>>> patch release.
> > > >>>>>>>>
> > > >>>>>>>> Personally speaking, I would love to do this, for exactly the
> > > >>>> reason
> > > >>>>>> you
> > > >>>>>>>> mentioned. However, I did not propose this due to the
> following
> > > >>>>>> reasons:
> > > >>>>>>>>
> > > >>>>>>>> 1. I am hesitating a little bit about changing the accepted
> > > >> FLIPs
> > > >>>> too
> > > >>>>>>> soon.
> > > >>>>>>>> 2. More importantly, to avoid slowing down our development. At
> > > >> this
> > > >>>>>>> point,
> > > >>>>>>>> Flink still lacks some design / routines to support good API
> > > >>>>>>> evolvability /
> > > >>>>>>>> extensibility. Just like you said, it takes some time to be
> > > >> good at
> > > >>>>>> this.
> > > >>>>>>>> In this case, my concern is that only removing Experimental /
> > > >>>>>>>> PublicEvolving APIs in major version changes may result in too
> > > >> much
> > > >>>>>>>> overhead and dramatically slow down the development of Flink.
> > > >> So, I
> > > >>>>> was
> > > >>>>>>>> thinking that we can start with the current status. Hopefully
> > > >> after
> > > >>>>> we
> > > >>>>>>> are
> > > >>>>>>>> more comfortable with the maintenance overhead of deprecated
> > > >> APIs,
> > > >>>> we
> > > >>>>>> can
> > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > >> PublicEvolving
> > > >>>>> APIs.
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>>
> > > >>>>>>>> Jiangjie (Becket) Qin
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > >> vvcephei@apache.org <ma...@apache.org>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Becket,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really
> > > >>>>>> important.
> > > >>>>>>> I
> > > >>>>>>>>> understand some people’s concerns about the additional burden
> > > >> for
> > > >>>>>>> project
> > > >>>>>>>>> maintainers, but my personal experience with Kafka has been
> > > >> that
> > > >>>>> it’s
> > > >>>>>>> very
> > > >>>>>>>>> liveable and that it’s well worth the benefit to users. In
> > > >> fact,
> > > >>>>> users
> > > >>>>>>>>> being able to confidently upgrade is also a benefit to
> > > >>>> maintainers,
> > > >>>>> as
> > > >>>>>>> we
> > > >>>>>>>>> will get fewer questions from people stuck on very old
> > > >> versions.
> > > >>>>>>>>>
> > > >>>>>>>>> One question:
> > > >>>>>>>>> Can we consider only dropping deprecated APIs in major
> > > >> releases
> > > >>>>> across
> > > >>>>>>> the
> > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> > > >> are
> > > >>>> by
> > > >>>>>>>>> definition less stable, but it seems like this should be
> > > >> reflected
> > > >>>>> in
> > > >>>>>>> the
> > > >>>>>>>>> required deprecation period alone. I.e. that we must keep
> them
> > > >>>>> around
> > > >>>>>>> for
> > > >>>>>>>>> at least zero or one minor release, not that we can drop them
> > > >> in a
> > > >>>>>>> minor or
> > > >>>>>>>>> patch release.
> > > >>>>>>>>>
> > > >>>>>>>>> The advantage of forbidding the removal of any API in minor
> or
> > > >>>> patch
> > > >>>>>>>>> releases is that users will get a strong guarantee that they
> > > >> can
> > > >>>>> bump
> > > >>>>>>> the
> > > >>>>>>>>> minor or patch version and still be able to compile, or even
> > > >> just
> > > >>>>>>> re-link
> > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run
> > > >> time.
> > > >>>>> This
> > > >>>>>>> is a
> > > >>>>>>>>> binary guarantee: if we allow removing  even Experimental
> APIs
> > > >>>>> outside
> > > >>>>>>> of
> > > >>>>>>>>> major releases, users can no longer confidently upgrade.
> > > >>>>>>>>>
> > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points:
> > > >>>>>>>>> * I’d use the official Deprecated annotation instead of
> > > >>>> introducing
> > > >>>>>> our
> > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> integrated
> > > >>>> into
> > > >>>>>>> build
> > > >>>>>>>>> tools and IDEs.
> > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it
> > > >> seems
> > > >>>>>>>>> orthogonal, and something that should probably be taken
> > > >>>> case-by-case
> > > >>>>>>>>> anyway.
> > > >>>>>>>>> * Aside from deprecation and removal, there have been some
> > > >>>>> discussions
> > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways.
> > > >> This is
> > > >>>>>>> somewhat
> > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before,
> it’ll
> > > >>>> take
> > > >>>>>> some
> > > >>>>>>>>> time to become good at it. I feel like this topic should also
> > > >> be
> > > >>>>>>> orthogonal
> > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
> > > >> simple
> > > >>>>>> policy
> > > >>>>>>> not
> > > >>>>>>>>> to break existing user programs, and leave the “how” up to
> > > >>>>>> implementers
> > > >>>>>>> and
> > > >>>>>>>>> reviewers.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks again,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > >>>>>>>>>> Hi All,
> > > >>>>>>>>>>
> > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
> > > >> great
> > > >>>>> idea.
> > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > >>>> @PublicEvolving
> > > >>>>>> --(2
> > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > > >> flexibility
> > > >>>>>>> without
> > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> > > >> change
> > > >>>>>> between
> > > >>>>>>>>> major
> > > >>>>>>>>>> releases. Changing annotations is acceptable and provides
> > > >>>>> additional
> > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself
> > > >> are
> > > >>>> not
> > > >>>>>>>>> changed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I
> > > >> want to
> > > >>>>>> move
> > > >>>>>>> one
> > > >>>>>>>>>> step further and suggest introducing one more annotation
> > > >>>> @Retired.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > > >> downgrading
> > > >>>>>>> @Public
> > > >>>>>>>>> to
> > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java
> > > >>>> standard
> > > >>>>>>>>>> @deprecated should be used in the early stage of the
> > > >> deprecation
> > > >>>>>>> process
> > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does not
> > > >>>> allow
> > > >>>>>> us
> > > >>>>>>> to
> > > >>>>>>>>>> extend annotation, I think it would be feasible to have the
> > > >> new
> > > >>>>>>> @Retired
> > > >>>>>>>>> to
> > > >>>>>>>>>> help us monitor and manage the deprecation process, house
> > > >>>>> cleaning,
> > > >>>>>>> etc.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Retired:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > >> compatibility
> > > >>>>>> before
> > > >>>>>>>>> APIs
> > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes
> > > >> on
> > > >>>> the
> > > >>>>> ML
> > > >>>>>>> are
> > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
> > > >> release
> > > >>>>>>> sharply
> > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether it
> > > >> is
> > > >>>>>>> possible).
> > > >>>>>>>>>> Devs who marked them as @Retired are responsible to remove
> > > >> them.
> > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
> > > >> are
> > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
> > > >> @Retired
> > > >>>>>> could
> > > >>>>>>>>> also
> > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > >>>> dev-friendliness,
> > > >>>>>> but
> > > >>>>>>> not
> > > >>>>>>>>>> mandatory.
> > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support the
> > > >>>>>>> deprecation
> > > >>>>>>>>>> process management. Further extension is possible, since the
> > > >>>>>>> annotation
> > > >>>>>>>>> is
> > > >>>>>>>>>> built by us.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best regards,
> > > >>>>>>>>>> Jing
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > > >>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi Xintong,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for the explanation. Please see the replies inline
> > > >>>> below.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public API
> > > >> is
> > > >>>>> also a
> > > >>>>>>>>> kind of
> > > >>>>>>>>>>>> such change, just like removing one, which can only
> > > >> happen
> > > >>>> with
> > > >>>>>>> major
> > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> > > >>>> APIs
> > > >>>>>>>>> anytime,
> > > >>>>>>>>>>> but
> > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > >> reaching
> > > >>>> the
> > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > >> would
> > > >>>> be a
> > > >>>>>>> weaker
> > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > >> major
> > > >>>>>> version
> > > >>>>>>>>> bumps,
> > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> > > >>>>>> migration
> > > >>>>>>>>>>> period.
> > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > >> sounds
> > > >>>>>>> against
> > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > >> necessary,
> > > >>>>>> e.g.,
> > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> > > >> change my
> > > >>>>> code
> > > >>>>>>> at
> > > >>>>>>>>> all,
> > > >>>>>>>>>>> my expectation is the following:
> > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> > > >> fine
> > > >>>>>> with
> > > >>>>>>> all
> > > >>>>>>>>>>> the 2.x versions.
> > > >>>>>>>>>>> Actually I think there are some dependency version
> > > >> resolution
> > > >>>>>>> policies
> > > >>>>>>>>> out
> > > >>>>>>>>>>> there which picks the highest minor version when the
> > > >>>> dependencies
> > > >>>>>>> pull
> > > >>>>>>>>> in
> > > >>>>>>>>>>> multiple minor versions of the same jar, which may be
> > > >> broken if
> > > >>>>> we
> > > >>>>>>>>> remove
> > > >>>>>>>>>>> the API in minor releases.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that
> > > >> we
> > > >>>>> bump
> > > >>>>>> up
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > >> However,
> > > >>>>> this
> > > >>>>>>> also
> > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> > > >>>> stay
> > > >>>>>>> stable
> > > >>>>>>>>>>>> within the major release series, as the series can end
> > > >>>> anytime.
> > > >>>>>>> IMHO,
> > > >>>>>>>>>>> short
> > > >>>>>>>>>>>> major release series are not something "make the end
> > > >> users
> > > >>>>>> happy",
> > > >>>>>>> but
> > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > >> frequent
> > > >>>>>> breaking
> > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > > >> can
> > > >>>>>> still
> > > >>>>>>>>> have
> > > >>>>>>>>>>>> longer major release series, while only allowing Public
> > > >> APIs
> > > >>>>>>>>> deprecated
> > > >>>>>>>>>>> at
> > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > >> the
> > > >>>> next
> > > >>>>>>> major
> > > >>>>>>>>>>>> version.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I totally agree that frequent major version bumps are not
> > > >>>> ideal,
> > > >>>>>> but
> > > >>>>>>>>> here
> > > >>>>>>>>>>> we are comparing it with a minor version bump which
> > > >> removes a
> > > >>>>>> Public
> > > >>>>>>>>> API.
> > > >>>>>>>>>>> So the context is that we have already decided to remove
> > > >> this
> > > >>>>>> Public
> > > >>>>>>> API
> > > >>>>>>>>>>> while keeping everything else backwards compatible. I
> > > >> think a
> > > >>>>> major
> > > >>>>>>>>> version
> > > >>>>>>>>>>> bump is a commonly understood signal here, compared with a
> > > >>>> minor
> > > >>>>>>> version
> > > >>>>>>>>>>> change. From end users' perspective, for those who are not
> > > >>>>>> impacted,
> > > >>>>>>> in
> > > >>>>>>>>>>> this case upgrading a major version is not necessarily more
> > > >>>>>> involved
> > > >>>>>>>>> than
> > > >>>>>>>>>>> upgrading a minor version - both should be as smooth as a
> > > >>>>>> dependency
> > > >>>>>>>>>>> version change. For those who are impacted, they will lose
> > > >> the
> > > >>>>>> Public
> > > >>>>>>>>> API
> > > >>>>>>>>>>> anyways and a major version bump ensures there is no
> > > >> surprise.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > > >>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Public API is a well defined common concept, and one of
> > > >> its
> > > >>>>>>>>>>>>> convention is that it only changes with a major version
> > > >>>>> change.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public
> > > >> API is
> > > >>>>>> also a
> > > >>>>>>>>> kind
> > > >>>>>>>>>>>> of such change, just like removing one, which can only
> > > >> happen
> > > >>>>>> with
> > > >>>>>>>>> major
> > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> > > >>>> APIs
> > > >>>>>>>>> anytime,
> > > >>>>>>>>>>> but
> > > >>>>>>>>>>>> only in the case major version bumps happen before
> > > >> reaching
> > > >>>> the
> > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > > >> would
> > > >>>> be a
> > > >>>>>>> weaker
> > > >>>>>>>>>>>> change compared to removing the API immediately upon
> > > >> major
> > > >>>>>> version
> > > >>>>>>>>> bumps,
> > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> > > >>>>>> migration
> > > >>>>>>>>>>> period.
> > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > > >> sounds
> > > >>>>>>> against
> > > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > > >> necessary,
> > > >>>>>> e.g.,
> > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> But it should be
> > > >>>>>>>>>>>>> completely OK to bump up the major version if we really
> > > >> want
> > > >>>>> to
> > > >>>>>>> get
> > > >>>>>>>>> rid
> > > >>>>>>>>>>> of
> > > >>>>>>>>>>>>> a public API, right?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > > >> that we
> > > >>>>>> bump
> > > >>>>>>> up
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > > >> However,
> > > >>>>> this
> > > >>>>>>> also
> > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> > > >>>> stay
> > > >>>>>>> stable
> > > >>>>>>>>>>>> within the major release series, as the series can end
> > > >>>> anytime.
> > > >>>>>>> IMHO,
> > > >>>>>>>>>>> short
> > > >>>>>>>>>>>> major release series are not something "make the end
> > > >> users
> > > >>>>>> happy",
> > > >>>>>>> but
> > > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > > >> frequent
> > > >>>>>> breaking
> > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > > >> can
> > > >>>>>> still
> > > >>>>>>>>> have
> > > >>>>>>>>>>>> longer major release series, while only allowing Public
> > > >> APIs
> > > >>>>>>>>> deprecated
> > > >>>>>>>>>>> at
> > > >>>>>>>>>>>> the end of the previous major version to be removed in
> > > >> the
> > > >>>> next
> > > >>>>>>> major
> > > >>>>>>>>>>>> version.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > >> (1-2
> > > >>>>> years)
> > > >>>>>>> to
> > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> > > >>>> again
> > > >>>>>> to
> > > >>>>>>> the
> > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> > > >>>> when
> > > >>>>>>>>> breaking
> > > >>>>>>>>>>>>> changes can be made.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
> > > >> for new
> > > >>>>>> major
> > > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > > >> needed.
> > > >>>> That
> > > >>>>>>> makes
> > > >>>>>>>>> 1-2
> > > >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving
> > > >> us
> > > >>>> more
> > > >>>>>>> time
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> prepare the major release if needed.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Xintong
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
> > > >>>>> versions
> > > >>>>>>> for
> > > >>>>>>>>>>>>> previous
> > > >>>>>>>>>>>>>> major versions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> > > >> viable
> > > >>>> that
> > > >>>>>> is
> > > >>>>>>> in
> > > >>>>>>>>>>>>> practice.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > > >> support
> > > >>>> for
> > > >>>>>> Java
> > > >>>>>>>>> 8/11,
> > > >>>>>>>>>>>>> which may very well be a problem for our current users.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
> > > >> about
> > > >>>>>> Public
> > > >>>>>>>>> API
> > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
> > > >> in
> > > >>>> the
> > > >>>>>>> NOTES
> > > >>>>>>>>> part
> > > >>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
> > > >> it.
> > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
> > > >> are on
> > > >>>>> the
> > > >>>>>>> same
> > > >>>>>>>>>>>>> page on that note.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > >>>>>>>>>>>>>> But it should be
> > > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > > >> really
> > > >>>> want
> > > >>>>>> to
> > > >>>>>>> get
> > > >>>>>>>>>>> rid
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>> a public API, right?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get us
> > > >> to
> > > >>>>> 2.0.
> > > >>>>>> ;)
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> There's a separate discussion to be had on the cadence
> > > >> of
> > > >>>>> major
> > > >>>>>>>>> releases
> > > >>>>>>>>>>>>> going forward, and there seem to be different opinions
> > > >> on
> > > >>>>> that.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between
> > > >>>> major
> > > >>>>>>> ones,
> > > >>>>>>>>>>>>> that for us means that users have to potentially deal
> > > >> with
> > > >>>>>>> breaking
> > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > > >> (1-2
> > > >>>>>> years)
> > > >>>>>>> to
> > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> > > >>>> again
> > > >>>>>> to
> > > >>>>>>> the
> > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> > > >>>> when
> > > >>>>>>>>> breaking
> > > >>>>>>>>>>>>> changes can be made.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
> > > >> Not
> > > >>>>> sure
> > > >>>>>> I
> > > >>>>>>>>> fully
> > > >>>>>>>>>>>>>> understand the concern for behavioral change here.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> This could be a lot of things. It can be performance in
> > > >>>>> certain
> > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
> > > >> relied
> > > >>>>> upon
> > > >>>>>>>>>>>>> (
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > ),
> > > a semantic change to some
> > > >> API.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> For a concrete example, consider the job submission. A
> > > >> few
> > > >>>>>>> releases
> > > >>>>>>>>> back
> > > >>>>>>>>>>>>> we made changes such that the initialization of the job
> > > >>>> master
> > > >>>>>>>>> happens
> > > >>>>>>>>>>>>> asynchronously.
> > > >>>>>>>>>>>>> This meant the job submission call returns sooner, and
> > > >> the
> > > >>>> job
> > > >>>>>>> state
> > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but the
> > > >>>>> observed
> > > >>>>>>>>> behavior
> > > >>>>>>>>>>>>> may be different.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Metrics are another example; I believe over time we
> > > >> changed
> > > >>>>> what
> > > >>>>>>> some
> > > >>>>>>>>>>>>> metrics returned a few times.
> > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Thanks much for the input, John, Stefan and Jing.

I think Xingtong has well summarized the pros and cons of the two options.
Let's collect a few more opinions here and we can move forward with the one
more people prefer.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi all,
>
> Thanks Xingtong for the summary. If I could only choose one of the given
> two options, I would go with option 1. I understood that option 2 worked
> great with Kafka. But the bridge release will still confuse users and my
> gut feeling is that many users will skip 2.0 and be waiting for 3.0 or even
> 3.x. And since fewer users will use Flink 2.x, the development focus will
> be on Flink 3.0 with the fact that the current Flink release is 1.17 and we
> are preparing 2.0 release. That is weird for me.
>
> THB, I would not name the change from @Public to @Retired as a demotion.
> The purpose of @Retire is to extend the API lifecycle with one more stage,
> like in the real world, people born, studied, graduated, worked, and
> retired. Afaiu from the previous discussion, there are two rules we'd like
> to follow simultaneously:
>
> 1. Public APIs can only be changed between major releases.
> 2. A smooth migration phase should be offered to users, i.e. at least 2
> minor releases after APIs are marked as @deprecated. There should be new
> APIs as the replacement.
>
> Agree, those rules are good to improve the user friendliness. Issues we
> discussed are rising because we want to fulfill both of them. If we take
> care of deprecation very seriously, APIs can be marked as @Deprecated, only
> when the new APIs as the replacement provide all functionalities the
> deprecated APIs have. In an ideal case without critical bugs that might
> stop users adopting the new APIs. Otherwise the expected "replacement" will
> not happen. Users will still stick to the deprecated APIs, because the new
> APIs can not be used. For big features, it will need at least 4 minor
> releases(ideal case), i.e. 2+ years to remove deprecated APIs:
>
> - 1st minor release to build the new APIs as the replacement and waiting
> for feedback. It might be difficult to mark the old API as deprecated in
> this release, because we are not sure if the new APIs could cover 100%
> functionalities.
> -  In the lucky case,  mark all old APIs as deprecated in the 2nd minor
> release. (I would even suggest having the new APIs released at least for
> two minor releases before marking it as deprecated to make sure they can
> really replace the old APIs, in case we care more about smooth migration)
> - 3rd minor release for the migration period
> -  In another lucky case, the 4th release is a major release, the
> deprecated APIs could be removed.
>
> The above described scenario works only in an ideal case. In reality, it
> might take longer to get the new APIs ready and mark the old API
> deprecated. Furthermore, if the 4th release is not a major release, we will
> have to maintain both APIs for many further minor releases. The question is
> how to know the next major release in advance, especially 4 minor releases'
> period, i.e. more than 2 years in advance? Given that Flink contains many
> modules, it is difficult to ask devs to create a 2-3 years deprecation plan
> for each case. In case we want to build major releases at a fast pace,
> let's say every two years, it means devs must plan any API deprecation
> right after each major release. Afaiac, it is quite difficult.
>
> The major issue is, afaiu, if we follow rule 2, we have to keep all @Public
> APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0. Then
> we have to follow rule 1 to keep it unchanged until we have 3.0. That is
> why @Retired is useful to give devs more flexibility and still fulfill both
> rules. Let's check it with examples:
>
> - we have @Public DataStream API in 1.18. It will not be marked
> as @Deprecated, because the new APIs as the replacement are not ready.
> - we keep the DataStream API itself unchanged in 2.0, but change the
> annotation from @Public to @Retire. New APIs will be introduced too. In
> this case, Rule 1 is ok, since public API is allowed to change between
> major releases. Only changing annotation is the minimal change we could do
> and it does not break rule 1. Rule 2 is ok too, since the DataStream APIs
> work exactly the same.  Attention: the change of @Public -> @Retired can
> only be done between major releases, because @Public APIs can only be
> changed between major releases.
> - in 2.1, DataStream API will be marked as deprecated.
> - in 2.2, DataStream will be kept for the migration period.
> - in 2.3, DataStream will be removed.
>
> Becket mentioned previously (please correct me if I didn't understand it
> correctly) that users might not check the changes of annotation and the
> upgrade process might not be smooth. I was wondering, if users don't pay
> attention to that, they will also not pay attention to @deprecated. The
> migration will not be smooth too even when we stick to the Java
> standard @deprecated. Let's go throw it with examples:
>
> Scenario 1: with @Retired
>
> Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be done
> for users who still want to use DataStream. The migration is smooth. But
> they should be aware of the annotation changes, i.e. @Public -> @Retired
> Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since it is
> a major release upgrade, breaking changes are expected.
> Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed. Since APIs
> are marked as @Retired, breaking changes are expected too.
>
> In addition to fulfilling both rules, there are two more benefits of this
> option:
> 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> 2. no bridge release is required. DataStream API could be deprecated in 2.x
> without breaking any of the two rules. @Retired provides enough flexibility
> for future deprecations and upgrades.
>
> The only issue I can find is that users might be surprised in case 3, if
> they ignore the previous @Public -> @Retired change. But for such users,
> they will ignore @deprecated annotation too and will have issues with
> pure @deprecated
> annotation too:
>
> Scenario 2: with @deprecated
>
> The example will be slightly different from scenario 1, because @Public
> APIs can only be removed between major releases. As an example, DataStream
> API will be marked as deprecated in 1.19, kept in 1.20, and then removed in
> 2.0
>
> Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as
> deprecated are removed between major releases. Rule 1 is ok. The 2 minor
> releases period defined in rule 2 sounds like a smooth migration plan, but
> for users in this case, it is still not smooth.
> Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with
> deprecated APIs. But for those users who ignore annotation changes, will
> have the same issue mentioned in case 1.
>
> I understood that, compared to scenario 1, users, who ignore @Public
> -> @Retired
> changes, will not face any breaking changes, after they migrated to 2.0.
> But the cost of it is bridge releases. There might be many bridge releases
> in the future. As I mentioned previously, it is not easy to find the right
> timing to kick off the deprecation process of APIs before the next expected
> major release and 2-3 years in advance.
>
> Scenario 3: @PublicEvolving deprecation for users who ignore @deprecated
> annotation changes
>
> Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21
>
> Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth
> migration.
> Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> since @deprecated is ignored
>
> In summary, for users who care about the annotation change, they will get
> more benefits with @Retired. For users who ignore annotation changes, they
> will be always facing non-smooth upgrades even using pure standard
> @deprecated
> annotation/process.
>
> For the case that IDEs support @deprecated, that is true. There are no such
> supports for homebuilt annotation. But the intention is to let users be
> aware of it. There will be other ways to do it for users who care about
> annotation changes. For users who do not care, there should be no
> difference between with tool support or without. Using @Retired with
> @deprecation
> together is at least not worse than only using @deprecation alone. And
> there are additional benefits with @Retired.
>
> Just my 2 cents and looking forward to your feedback, especially different
> opinions that will help me understand the issue better. Thanks!
>
>
> Best regards,
> Jing
>
>
>
>
>
>
> On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> <sr...@confluent.io.invalid> wrote:
>
> > Hi Xintong,
> >
> > Thanks for the summary, most of the points that you agreed upon also make
> > sense to me.
> >
> > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> APIs
> > >   from Public to PublicEvolving / Experimental / Retired in any version
> > bump,
> > >   are not good practices.
> >
> > I hope we can can go beyond calling it “not a good practice" and reach
> > consensus that we will not demote or remove public APIs in minor releases
> > and (technically) enforce this rule for all contributions. I think it’s a
> > reasonable expectation for stability from a project as mature as Flink.
> >
> > > I'm personally in favor of option 1. As the migration-period rule is
> > newly
> > > proposed, I think it's fair to make exceptions for cases where we
> already
> > > missed the best chance for planning the deprecation. Moreover, I do
> > believe
> > > having a quick major version bump does confuse users. Yes, we can
> explain
> > > to users that bumping from 2.x to 3.0 does not cost anything other than
> > the
> > > removal of an deprecated API. But having to explain this itself is an
> > > indicator that it might be confusing for users.
> > >
> > >
> > > Becket, on the other hand, prefers option 2. From my understanding, his
> > > major point is that a quick major version bump causes barely any actual
> > > lose on users, while in option 1 not providing the migration period or
> > > providing a shorter on is an actual lose on users. (@Becket, please
> > correct
> > > me if I'm mistaken.)
> > >
> >
> > I would be open to both options and maybe postpone this decision until we
> > have the complete roadmap for Flink 2.0. We would have a better overview
> > about the technical consequences, e.g. how hard it would be to offer and
> > maintain both APIs side by side, how well we get both APIs separated, and
> > whether or not we will be able to use the full potential of our breaking
> > changes before the DataStream API is completely removed. One example for
> > the last point I can think of would be moving from lazy to eager state
> > declaration. If we believe that it is reasonable to support all planned
> > changes while maintaining both APIs, I'm leaning towards option 2.
> >
> > Best,
> > Stefan
> >
> >
> > > On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com> wrote:
> > >
> > > Becket and I had an offline voice call earlier today. We have reached
> > > consensus on some of the arguments, but still hold different opinions
> on
> > > some others. I'd like to post the outcome here for transparency.
> > >
> > > We both agree that:
> > >
> > >   1. Providing a migration period would be beneficial for our users,
> and
> > >   we should do that
> > >   2. Dropping deprecated Public APIs in minor releases, or demoting
> APIs
> > >   from Public to PublicEvolving / Experimental / Retired in any version
> > bump,
> > >   are not good practices.
> > >   3. Ideally, with this FLIP, developers should be more careful and
> plan
> > >   API changes ahead. That means:
> > >      1. Be more careful with designing APIs and promoting them to
> Public,
> > >      so that they won't be changed / removed very soon, and also the
> > >      maintenance overhead for keeping them after deprecation should be
> > >      affordable. (I believe this also aligns with John's opinion.)
> > >      2. Plan the deprecation / removal of old APIs that we don't want
> to
> > >      carry to the next major version earlier, so that they'll have the
> > >      2-minor-release migration period before the next major version
> bump.
> > >   4. A practical situation is that, we are not ready for deprecating
> the
> > >   DataStream API, nor afford to carry it for another couple of years.
> > >      1. We cannot deprecate it now, because the new ProcessFunction API
> > >      (the planned replacement for DataStream API) is not yet ready. We
> > haven't
> > >      planned the ProcessFunction API earlier because the original
> > > plan was to do
> > >      in-place changes directly on the DataStream API in release 2.0,
> > until the
> > >      smooth migration period is proposed. Before this, the only
> > guarantees we
> > >      provide for Public APIs is that they'll stay compatible until the
> > next
> > >      major version, which led to the understanding the in-place changes
> > on
> > >      Public APIs should be fine.
> > >      2. We also cannot afford carrying the DataStream API for another
> > >      couple of years. One of the reasons that we want to refactor /
> > replace
> > >      DataStream API is that it exposes and depends on too many
> > > Flink's internal
> > >      runtime implementations, which has already limited / blocked plans
> > for
> > >      Flink's internal improvements for many times. Also due to the wide
> > and
> > >      transitive dependencies on the internals, maintaining two sets of
> > lower
> > >      APIs itself is expensive and would be better to keep it short.
> This
> > is
> > >      admittedly a historical design flaw which should be avoided in
> > future.
> > >
> > > What we haven't reached consensus is about how to deal with the current
> > > situation around the DataStream API. There are two options.
> > >
> > >
> > >   1. We say this FLIP is enforced starting release 2.0. For current 1.x
> > >   APIs, we provide a migration period with best effort, while allowing
> > >   exceptions for immediate removal in 2.0. That means we will still try
> > with
> > >   best effort to get the ProcessFuncion API ready and deprecate the
> > >   DataStream API in 1.x, but will also be allowed to remove DataStream
> > API in
> > >   2.0 if it's not deprecated 2 minor releases before the major version
> > bump.
> > >   2. We strictly follow the process in this FLIP, and will quickly bump
> > >   the major version from 2.x to 3.0 once the migration period for
> > DataStream
> > >   API is reached.
> > >
> > > I'm personally in favor of option 1. As the migration-period rule is
> > newly
> > > proposed, I think it's fair to make exceptions for cases where we
> already
> > > missed the best chance for planning the deprecation. Moreover, I do
> > believe
> > > having a quick major version bump does confuse users. Yes, we can
> explain
> > > to users that bumping from 2.x to 3.0 does not cost anything other than
> > the
> > > removal of an deprecated API. But having to explain this itself is an
> > > indicator that it might be confusing for users.
> > >
> > >
> > > Becket, on the other hand, prefers option 2. From my understanding, his
> > > major point is that a quick major version bump causes barely any actual
> > > lose on users, while in option 1 not providing the migration period or
> > > providing a shorter on is an actual lose on users. (@Becket, please
> > correct
> > > me if I'm mistaken.)
> > >
> > >
> > > And we'd like to hear more feedback from the community.
> > >
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcephei@apache.org
> > <ma...@apache.org>> wrote:
> > >
> > >> Hi Becket and Xintong,
> > >>
> > >> I hope you don't mind if I chime in a little.
> > >>
> > >> Once an API is marked Public, we're committing to support it until
> it's
> > >> deprecated, and once it's deprecated, to leave it in place for at
> least
> > two
> > >> minor releases and then only remove it in the following major release.
> > >>
> > >> As a user, I would be dismayed to discover that the project
> maintainers
> > >> demoted a Public API to PublicEvolving or Experimental in order to
> > violate
> > >> the spirit of the deprecation period for Public APIs. Likewise, I'd
> > view a
> > >> rapid sequence of minor releases for the purpose of dropping
> deprecated
> > >> APIs in the next major release as another violation of the spirit of
> our
> > >> guarantees. I'm reminded of this xkcd:
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > >>
> > >> I'm glad we're gaming out these situations before we institute this
> > FLIP,
> > >> and I hope we all converge on a clear understanding of what we're
> > >> guaranteeing. From where I'm sitting, the essence of this FLIP is
> > >> specifically to prevent us from doing whatever we want and instead to
> > >> commit to providing a suitable notice period to users who rely on
> Public
> > >> APIs.
> > >>
> > >> It honestly shouldn't matter what the maintenance overhead is (or,
> > rather,
> > >> evaluating the maintenance overhead should be a consideration before
> we
> > >> promote an API to Public to begin with). My experience in other
> > projects is
> > >> that regretting a feature so much you want to drop it asap is
> extremely
> > >> rare, and that living with the pain for a little while until the
> > >> deprecation period expires will probably make us better engineers.
> > >>
> > >> With that all said, one thing that is not a violation is to propose a
> > new
> > >> major release in order to drop especially burdensome deprecated APIs
> > that
> > >> have already enjoyed their full deprecation period. Those really bad
> > "case
> > >> A" features will be very rare.
> > >>
> > >> Looking over the FLIP again, I see a reference to the release
> > >> documentation, but not to the build tooling support for the Deprecated
> > >> annotation, so it might help to share this scenario: The way this will
> > all
> > >> come together in practice is that, by giving good stability
> guarantees,
> > we
> > >> will encourage our users to bump minor versions when they are
> released,
> > >> which means that they will see compiler warnings immediately if any of
> > the
> > >> APIs they've used are deprecated. In fact, I'd recommend for them to
> use
> > >> the `-Werror` javac flag to be sure that they don't miss those
> warnings.
> > >> Given our release cadence, this means that they will have about six
> > months
> > >> to migrate off of any deprecated APIs without being forced to avoid
> > >> updates. And they really need to be able to adopt further minor
> updates
> > in
> > >> the series, since they may contain further improvements that actually
> > >> facilitate migrating off of the deprecated APIs.
> > >>
> > >> The compiler warnings are a far more reliable mechanism to advertise
> > >> deprecation than release notes, since users' own builds will notify
> them
> > >> right away about only the deprecations that are relevant to them, but
> it
> > >> does rely on users feeling ok to bump up minor versions in a timely
> > >> fashion. If we consistently burn them by not observing the stability
> > >> guarantees, they'll try to avoid updates instead, and the whole thing
> > falls
> > >> apart.
> > >>
> > >> Thanks again,
> > >> -John
> > >>
> > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > >>> Hi Xintong,
> > >>>
> > >>> Let's compare the following cases:
> > >>>
> > >>> A. If the maintenance overhead of the deprecated API is high, and we
> > want
> > >>> to remove it after two minor releases. Then there are two options:
> > >>>    A1: Demote the API in the major version bump and remove the code
> > >> with a
> > >>> minor version bump.
> > >>>    A2: Do exactly the same as 1-A, except that when removing the
> code,
> > >>> bump up the major version. So this might be a short major version.
> > >>>
> > >>> B. If the maintenance overhead of the deprecated API is not high,
> > >> therefore
> > >>> keeping it for a long time is affordable. There are also two options:
> > >>>    B1: Same as A-1, demote the API in the major version bump and
> remove
> > >>> the code with a minor version bump.
> > >>>    B2: Keep the API for all the minor versions in the major version,
> > and
> > >>> only remove the code in the next major version.
> > >>>
> > >>> For case B, do we agree that B2 is the way to go?
> > >>>
> > >>> For case A:
> > >>> The following stays the same for A1 and A2:
> > >>>  - users will lose the API after two minor leases. So the migration
> > >> period
> > >>> is the same for A-1 and A-2.
> > >>>  - A1 and A2 will have exactly the same code after the removal of
> > >>> deprecated API, the only difference is versioning.
> > >>>  - To move forward, users need to move to the next minor version in
> A1,
> > >> or
> > >>> to the next major version in A2. Because the code are the same, the
> > >> actual
> > >>> effort to upgrade is the same for A1 and A2.
> > >>>
> > >>> The differences between A-1 and A-2 are:
> > >>>  - A1 allows keeping the major version release cadence. A-2 will
> have a
> > >>> short major version release.
> > >>>  - A1 breaks the well understood API semantic, while A-2 does not.
> > >>>
> > >>> From what I see, since there is no well establish standard regarding
> > how
> > >>> long a major version should be released, A short major version
> release
> > is
> > >>> potential and emotional. It is not ideal but does not have material
> > >>> downsides compared with A1. I did not hear anyone complaining about
> > Kafka
> > >>> only has two 1.x release. However, A1 actually breaks the well
> > understood
> > >>> API semantic, which has more material impact.
> > >>>
> > >>> Also, I'd imagine 90% or more of the Public APIs should fall into
> case
> > B.
> > >>> So, short major versions should be very occasional. I'd be very
> > concerned
> > >>> if the reason we choose A1 is simply because we cannot afford
> > >> maintaining a
> > >>> bunch of deprecated APIs until the next major version. This indicates
> > >> that
> > >>> the actual problem we need to solve is to lower the maintenance
> > overhead
> > >> of
> > >>> deprecated APIs, so that we are comfortable to keep them longer. As
> > John
> > >>> and I mentioned earlier, there are ways to achieve this and we need
> to
> > >>> learn how to do it in Flink. Otherwise, our discussion about
> versioning
> > >>> here does not bring much value, because we will end up with a bunch
> of
> > >>> short-lived APIs which upset our users, no matter how we version the
> > >>> releases.
> > >>>
> > >>> So, if there are concrete examples that you think will block us from
> > >>> keeping API stability with affordable cost, let's take a look
> together
> > >> and
> > >>> see if that can be improved.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jiangjie (Becket) Qin
> > >>>
> > >>>
> > >>> From what I see,
> > >>>
> > >>>
> > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong820@gmail.com
> > <ma...@gmail.com>>
> > >> wrote:
> > >>>
> > >>>>>
> > >>>>> The part I don't understand is if we are willing to have a
> migration
> > >>>>> period, and do a minor version bump to remove an API, what do we
> > >> lose to
> > >>>> do
> > >>>>> a major version bump instead, so we don't break the common
> versioning
> > >>>>> semantic?
> > >>>>>
> > >>>>
> > >>>> I think we are talking about the cases where a major version bump
> > >> happens
> > >>>> before a deprecated Public API reaches its migration period. So
> > >> removing
> > >>>> the API with another major version bump means we have two
> consecutive
> > >> major
> > >>>> versions in a very short time. And as previously mentioned, having
> > >> frequent
> > >>>> major version bumps would weaken the value of the commitment "Public
> > >> API
> > >>>> stay compatible within a major version". I think users also have
> > >>>> expectations about how long a major version should live, which
> should
> > >> be at
> > >>>> least a couple of years rather than 1-2 minor releases.
> > >>>>
> > >>>> This is another option, but I think it is very likely more expensive
> > >> than
> > >>>>> simply bumping the major version. And I imagine there will be
> > >> questions
> > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > >>>>>
> > >>>>
> > >>>> Yes, it's more expensive than simply bumping the major version, but
> > >> IMHO
> > >>>> it's probably cheaper than carrying the API until the next major
> > >> version if
> > >>>> we don't bump the major version very soon. And see my reply above on
> > >> why
> > >>>> not bumping immediately.
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>> Xintong
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket.qin@gmail.com
> > <ma...@gmail.com>>
> > >> wrote:
> > >>>>
> > >>>>> Hi Xintong,
> > >>>>>
> > >>>>> Please see the replies below.
> > >>>>>
> > >>>>> I see your point, that users would feel surprised if they find
> > >> things no
> > >>>>>> longer work when upgrading to another 2.x minor release. However,
> > >> I'd
> > >>>>> like
> > >>>>>> to point out that PublicEvolving APIs would have the similar
> > >> problem
> > >>>>>> anyway. So the question is, how do we catch users' attention and
> > >> make
> > >>>>> sure
> > >>>>>> they are aware that the Public APIs in 1.x may no longer be Public
> > >> in
> > >>>>> 2.0.
> > >>>>>> There're various ways to do that, e.g., release notes, warnings in
> > >>>> logs,
> > >>>>>> etc.
> > >>>>>
> > >>>>> First of all, I am not a fan of removing PublicEvolving APIs in
> minor
> > >>>>> version changes. Personally speaking, I want them to be also
> removed
> > >> in
> > >>>>> major version changes.
> > >>>>>
> > >>>>> Empirically, I rarely see projects with complex API semantic work
> > >> well.
> > >>>>> Many, if not most, users don't read docs / release notes / warnings
> > >> in
> > >>>>> logs. I don't think that is their fault. Typically an application
> > >>>> developer
> > >>>>> will have to deal with dozens of libraries maintained by various
> > >>>>> communities / groups. It is just too difficult for them to keep
> > >> track of
> > >>>>> all the specific semantics each project puts there. If you think
> > >> about
> > >>>> it,
> > >>>>> how many of Flink developers read all the release notes of Guava,
> > >> Apache
> > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of
> > >> them?
> > >>>>> One would probably try bumping the dependency version and see if
> > >> there is
> > >>>>> an exception and solve them case by case. And if it is a minor
> > >> version
> > >>>>> bump, one probably does not expect an exception at all. Another
> > >> example
> > >>>> is
> > >>>>> that in Flink we still have so many usage of deprecated methods all
> > >> over
> > >>>>> the place, and I strongly doubt everyone knows when the source code
> > >> of
> > >>>>> these methods are deprecated and when they should be removed.
> > >>>>>
> > >>>>> So, the most important principle of API is simple and intuitive.
> The
> > >>>>> versioning semantic is a simple and universally accepted API
> > >> stability
> > >>>>> standard. If we ourselves as Flink developers are relying on this
> > >> for our
> > >>>>> own dependencies. I don't think we can expect more from our users.
> > >>>>>
> > >>>>> Another possible alternative: whenever there's a deprecated Public
> > >> API
> > >>>> that
> > >>>>>> reaches a major version bump before the migration period, and we
> > >> also
> > >>>>> don't
> > >>>>>> want to carry it for all the next major release series, we may
> > >> consider
> > >>>>>> releasing more minor releases for the previous major version after
> > >> the
> > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
> > >> to
> > >>>> 2.0,
> > >>>>>> we can release one more 1.20 after 2.0. That should provide users
> > >>>> another
> > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > >>>> 2-minor-release
> > >>>>>> migration period.
> > >>>>>
> > >>>>> This is another option, but I think it is very likely more
> expensive
> > >> than
> > >>>>> simply bumping the major version. And I imagine there will be
> > >> questions
> > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> > >>>>>
> > >>>>> I think my major point is, we should not carry APIs deprecated in a
> > >>>>>> previous major version along all the next major version series.
> I'd
> > >>>> like
> > >>>>> to
> > >>>>>> try giving users more commitments, i.e. the migration period, as
> > >> long
> > >>>> as
> > >>>>> it
> > >>>>>> does not prevent us from making breaking changes. If it doesn't
> > >> work,
> > >>>> I'd
> > >>>>>> be in favor of not providing the migration period, but fallback to
> > >> only
> > >>>>>> guarantee the compatibility within the major version.
> > >>>>>
> > >>>>> The part I don't understand is if we are willing to have a
> migration
> > >>>>> period, and do a minor version bump to remove an API, what do we
> > >> lose to
> > >>>> do
> > >>>>> a major version bump instead, so we don't break the common
> versioning
> > >>>>> semantic?
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jiangjie (Becket) Qin
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> tonysong820@gmail.com
> > <ma...@gmail.com>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>>>
> > >>>>>>> As an end user who only uses Public APIs, if I don't change my
> > >> code
> > >>>> at
> > >>>>>>> all, my expectation is the following:
> > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > >> with
> > >>>>> all
> > >>>>>>> the 2.x versions.
> > >>>>>>> Actually I think there are some dependency version resolution
> > >>>> policies
> > >>>>>> out
> > >>>>>>> there which picks the highest minor version when the dependencies
> > >>>> pull
> > >>>>> in
> > >>>>>>> multiple minor versions of the same jar, which may be broken if
> > >> we
> > >>>>> remove
> > >>>>>>> the API in minor releases.
> > >>>>>>>
> > >>>>>>
> > >>>>>> I see your point, that users would feel surprised if they find
> > >> things
> > >>>> no
> > >>>>>> longer work when upgrading to another 2.x minor release. However,
> > >> I'd
> > >>>>> like
> > >>>>>> to point out that PublicEvolving APIs would have the similar
> > >> problem
> > >>>>>> anyway. So the question is, how do we catch users' attention and
> > >> make
> > >>>>> sure
> > >>>>>> they are aware that the Public APIs in 1.x may no longer be Public
> > >> in
> > >>>>> 2.0.
> > >>>>>> There're various ways to do that, e.g., release notes, warnings in
> > >>>> logs,
> > >>>>>> etc.
> > >>>>>>
> > >>>>>> Another possible alternative: whenever there's a deprecated Public
> > >> API
> > >>>>> that
> > >>>>>> reaches a major version bump before the migration period, and we
> > >> also
> > >>>>> don't
> > >>>>>> want to carry it for all the next major release series, we may
> > >> consider
> > >>>>>> releasing more minor releases for the previous major version after
> > >> the
> > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
> > >> to
> > >>>> 2.0,
> > >>>>>> we can release one more 1.20 after 2.0. That should provide users
> > >>>> another
> > >>>>>> choice rather than upgrading to 2.0, while satisfying the
> > >>>> 2-minor-release
> > >>>>>> migration period.
> > >>>>>>
> > >>>>>> I think my major point is, we should not carry APIs deprecated in
> a
> > >>>>>> previous major version along all the next major version series.
> I'd
> > >>>> like
> > >>>>> to
> > >>>>>> try giving users more commitments, i.e. the migration period, as
> > >> long
> > >>>> as
> > >>>>> it
> > >>>>>> does not prevent us from making breaking changes. If it doesn't
> > >> work,
> > >>>> I'd
> > >>>>>> be in favor of not providing the migration period, but fallback to
> > >> only
> > >>>>>> guarantee the compatibility within the major version.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>>
> > >>>>>> Xintong
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> vvcephei@apache.org
> > <ma...@apache.org>
> > >>>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Becket,
> > >>>>>>>
> > >>>>>>> Thanks for the reply! I’d like to continue the conversation about
> > >>>>>>> compatibility outside of this FLIP thread, but for now, I can
> > >> accept
> > >>>>> your
> > >>>>>>> decision. It’s certainly an improvement.
> > >>>>>>>
> > >>>>>>> Thanks again,
> > >>>>>>> John
> > >>>>>>>
> > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > >>>>>>>> Hi John,
> > >>>>>>>>
> > >>>>>>>> Completely agree with all you said.
> > >>>>>>>>
> > >>>>>>>> Can we consider only dropping deprecated APIs in major releases
> > >>>>> across
> > >>>>>>> the
> > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> > >> are
> > >>>> by
> > >>>>>>>>> definition less stable, but it seems like this should be
> > >> reflected
> > >>>>> in
> > >>>>>>> the
> > >>>>>>>>> required deprecation period alone. I.e. that we must keep them
> > >>>>> around
> > >>>>>>> for
> > >>>>>>>>> at least zero or one minor release, not that we can drop them
> > >> in a
> > >>>>>>> minor or
> > >>>>>>>>> patch release.
> > >>>>>>>>
> > >>>>>>>> Personally speaking, I would love to do this, for exactly the
> > >>>> reason
> > >>>>>> you
> > >>>>>>>> mentioned. However, I did not propose this due to the following
> > >>>>>> reasons:
> > >>>>>>>>
> > >>>>>>>> 1. I am hesitating a little bit about changing the accepted
> > >> FLIPs
> > >>>> too
> > >>>>>>> soon.
> > >>>>>>>> 2. More importantly, to avoid slowing down our development. At
> > >> this
> > >>>>>>> point,
> > >>>>>>>> Flink still lacks some design / routines to support good API
> > >>>>>>> evolvability /
> > >>>>>>>> extensibility. Just like you said, it takes some time to be
> > >> good at
> > >>>>>> this.
> > >>>>>>>> In this case, my concern is that only removing Experimental /
> > >>>>>>>> PublicEvolving APIs in major version changes may result in too
> > >> much
> > >>>>>>>> overhead and dramatically slow down the development of Flink.
> > >> So, I
> > >>>>> was
> > >>>>>>>> thinking that we can start with the current status. Hopefully
> > >> after
> > >>>>> we
> > >>>>>>> are
> > >>>>>>>> more comfortable with the maintenance overhead of deprecated
> > >> APIs,
> > >>>> we
> > >>>>>> can
> > >>>>>>>> then have a stronger guarantee for Experimental /
> > >> PublicEvolving
> > >>>>> APIs.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jiangjie (Becket) Qin
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > >> vvcephei@apache.org <ma...@apache.org>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Becket,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really
> > >>>>>> important.
> > >>>>>>> I
> > >>>>>>>>> understand some people’s concerns about the additional burden
> > >> for
> > >>>>>>> project
> > >>>>>>>>> maintainers, but my personal experience with Kafka has been
> > >> that
> > >>>>> it’s
> > >>>>>>> very
> > >>>>>>>>> liveable and that it’s well worth the benefit to users. In
> > >> fact,
> > >>>>> users
> > >>>>>>>>> being able to confidently upgrade is also a benefit to
> > >>>> maintainers,
> > >>>>> as
> > >>>>>>> we
> > >>>>>>>>> will get fewer questions from people stuck on very old
> > >> versions.
> > >>>>>>>>>
> > >>>>>>>>> One question:
> > >>>>>>>>> Can we consider only dropping deprecated APIs in major
> > >> releases
> > >>>>> across
> > >>>>>>> the
> > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> > >> are
> > >>>> by
> > >>>>>>>>> definition less stable, but it seems like this should be
> > >> reflected
> > >>>>> in
> > >>>>>>> the
> > >>>>>>>>> required deprecation period alone. I.e. that we must keep them
> > >>>>> around
> > >>>>>>> for
> > >>>>>>>>> at least zero or one minor release, not that we can drop them
> > >> in a
> > >>>>>>> minor or
> > >>>>>>>>> patch release.
> > >>>>>>>>>
> > >>>>>>>>> The advantage of forbidding the removal of any API in minor or
> > >>>> patch
> > >>>>>>>>> releases is that users will get a strong guarantee that they
> > >> can
> > >>>>> bump
> > >>>>>>> the
> > >>>>>>>>> minor or patch version and still be able to compile, or even
> > >> just
> > >>>>>>> re-link
> > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run
> > >> time.
> > >>>>> This
> > >>>>>>> is a
> > >>>>>>>>> binary guarantee: if we allow removing  even Experimental APIs
> > >>>>> outside
> > >>>>>>> of
> > >>>>>>>>> major releases, users can no longer confidently upgrade.
> > >>>>>>>>>
> > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points:
> > >>>>>>>>> * I’d use the official Deprecated annotation instead of
> > >>>> introducing
> > >>>>>> our
> > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well integrated
> > >>>> into
> > >>>>>>> build
> > >>>>>>>>> tools and IDEs.
> > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it
> > >> seems
> > >>>>>>>>> orthogonal, and something that should probably be taken
> > >>>> case-by-case
> > >>>>>>>>> anyway.
> > >>>>>>>>> * Aside from deprecation and removal, there have been some
> > >>>>> discussions
> > >>>>>>>>> about how to evolve APIs and behavior in compatible ways.
> > >> This is
> > >>>>>>> somewhat
> > >>>>>>>>> of an art, and if folks haven’t wrestled with it before, it’ll
> > >>>> take
> > >>>>>> some
> > >>>>>>>>> time to become good at it. I feel like this topic should also
> > >> be
> > >>>>>>> orthogonal
> > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
> > >> simple
> > >>>>>> policy
> > >>>>>>> not
> > >>>>>>>>> to break existing user programs, and leave the “how” up to
> > >>>>>> implementers
> > >>>>>>> and
> > >>>>>>>>> reviewers.
> > >>>>>>>>>
> > >>>>>>>>> Thanks again,
> > >>>>>>>>> John
> > >>>>>>>>>
> > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > >>>>>>>>>> Hi All,
> > >>>>>>>>>>
> > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
> > >> great
> > >>>>> idea.
> > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > >>>> @PublicEvolving
> > >>>>>> --(2
> > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> > >> flexibility
> > >>>>>>> without
> > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> > >> change
> > >>>>>> between
> > >>>>>>>>> major
> > >>>>>>>>>> releases. Changing annotations is acceptable and provides
> > >>>>> additional
> > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself
> > >> are
> > >>>> not
> > >>>>>>>>> changed.
> > >>>>>>>>>>
> > >>>>>>>>>> I had similar thoughts when I was facing those issues. I
> > >> want to
> > >>>>>> move
> > >>>>>>> one
> > >>>>>>>>>> step further and suggest introducing one more annotation
> > >>>> @Retired.
> > >>>>>>>>>>
> > >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> > >> downgrading
> > >>>>>>> @Public
> > >>>>>>>>> to
> > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java
> > >>>> standard
> > >>>>>>>>>> @deprecated should be used in the early stage of the
> > >> deprecation
> > >>>>>>> process
> > >>>>>>>>>> and doesn't really meet our requirement. Since Java does not
> > >>>> allow
> > >>>>>> us
> > >>>>>>> to
> > >>>>>>>>>> extend annotation, I think it would be feasible to have the
> > >> new
> > >>>>>>> @Retired
> > >>>>>>>>> to
> > >>>>>>>>>> help us monitor and manage the deprecation process, house
> > >>>>> cleaning,
> > >>>>>>> etc.
> > >>>>>>>>>>
> > >>>>>>>>>> Some ideas could be(open for discussion):
> > >>>>>>>>>>
> > >>>>>>>>>> @Retired:
> > >>>>>>>>>>
> > >>>>>>>>>> 1. There must be a replacement with functionality
> > >> compatibility
> > >>>>>> before
> > >>>>>>>>> APIs
> > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes
> > >> on
> > >>>> the
> > >>>>> ML
> > >>>>>>> are
> > >>>>>>>>>> mandatory (a FLIP is recommended).
> > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
> > >> release
> > >>>>>>> sharply
> > >>>>>>>>>> (using ArchUnit to force it, needs further check whether it
> > >> is
> > >>>>>>> possible).
> > >>>>>>>>>> Devs who marked them as @Retired are responsible to remove
> > >> them.
> > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
> > >> are
> > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
> > >> @Retired
> > >>>>>> could
> > >>>>>>>>> also
> > >>>>>>>>>> be used if it can increase user-friendliness or
> > >>>> dev-friendliness,
> > >>>>>> but
> > >>>>>>> not
> > >>>>>>>>>> mandatory.
> > >>>>>>>>>> 4. Some variables will be defined in @Retired to support the
> > >>>>>>> deprecation
> > >>>>>>>>>> process management. Further extension is possible, since the
> > >>>>>>> annotation
> > >>>>>>>>> is
> > >>>>>>>>>> built by us.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Best regards,
> > >>>>>>>>>> Jing
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > >>>> becket.qin@gmail.com <ma...@gmail.com>
> > >>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Xintong,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the explanation. Please see the replies inline
> > >>>> below.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I agree. And from my understanding, demoting a Public API
> > >> is
> > >>>>> also a
> > >>>>>>>>> kind of
> > >>>>>>>>>>>> such change, just like removing one, which can only
> > >> happen
> > >>>> with
> > >>>>>>> major
> > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> > >>>> APIs
> > >>>>>>>>> anytime,
> > >>>>>>>>>>> but
> > >>>>>>>>>>>> only in the case major version bumps happen before
> > >> reaching
> > >>>> the
> > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > >> would
> > >>>> be a
> > >>>>>>> weaker
> > >>>>>>>>>>>> change compared to removing the API immediately upon
> > >> major
> > >>>>>> version
> > >>>>>>>>> bumps,
> > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> > >>>>>> migration
> > >>>>>>>>>>> period.
> > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > >> sounds
> > >>>>>>> against
> > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > >> necessary,
> > >>>>>> e.g.,
> > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > >>>>>>>>>>>
> > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> > >> change my
> > >>>>> code
> > >>>>>>> at
> > >>>>>>>>> all,
> > >>>>>>>>>>> my expectation is the following:
> > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> > >> fine
> > >>>>>> with
> > >>>>>>> all
> > >>>>>>>>>>> the 2.x versions.
> > >>>>>>>>>>> Actually I think there are some dependency version
> > >> resolution
> > >>>>>>> policies
> > >>>>>>>>> out
> > >>>>>>>>>>> there which picks the highest minor version when the
> > >>>> dependencies
> > >>>>>>> pull
> > >>>>>>>>> in
> > >>>>>>>>>>> multiple minor versions of the same jar, which may be
> > >> broken if
> > >>>>> we
> > >>>>>>>>> remove
> > >>>>>>>>>>> the API in minor releases.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that
> > >> we
> > >>>>> bump
> > >>>>>> up
> > >>>>>>>>> the
> > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > >> However,
> > >>>>> this
> > >>>>>>> also
> > >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> > >>>> stay
> > >>>>>>> stable
> > >>>>>>>>>>>> within the major release series, as the series can end
> > >>>> anytime.
> > >>>>>>> IMHO,
> > >>>>>>>>>>> short
> > >>>>>>>>>>>> major release series are not something "make the end
> > >> users
> > >>>>>> happy",
> > >>>>>>> but
> > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > >> frequent
> > >>>>>> breaking
> > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > >> can
> > >>>>>> still
> > >>>>>>>>> have
> > >>>>>>>>>>>> longer major release series, while only allowing Public
> > >> APIs
> > >>>>>>>>> deprecated
> > >>>>>>>>>>> at
> > >>>>>>>>>>>> the end of the previous major version to be removed in
> > >> the
> > >>>> next
> > >>>>>>> major
> > >>>>>>>>>>>> version.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I totally agree that frequent major version bumps are not
> > >>>> ideal,
> > >>>>>> but
> > >>>>>>>>> here
> > >>>>>>>>>>> we are comparing it with a minor version bump which
> > >> removes a
> > >>>>>> Public
> > >>>>>>>>> API.
> > >>>>>>>>>>> So the context is that we have already decided to remove
> > >> this
> > >>>>>> Public
> > >>>>>>> API
> > >>>>>>>>>>> while keeping everything else backwards compatible. I
> > >> think a
> > >>>>> major
> > >>>>>>>>> version
> > >>>>>>>>>>> bump is a commonly understood signal here, compared with a
> > >>>> minor
> > >>>>>>> version
> > >>>>>>>>>>> change. From end users' perspective, for those who are not
> > >>>>>> impacted,
> > >>>>>>> in
> > >>>>>>>>>>> this case upgrading a major version is not necessarily more
> > >>>>>> involved
> > >>>>>>>>> than
> > >>>>>>>>>>> upgrading a minor version - both should be as smooth as a
> > >>>>>> dependency
> > >>>>>>>>>>> version change. For those who are impacted, they will lose
> > >> the
> > >>>>>> Public
> > >>>>>>>>> API
> > >>>>>>>>>>> anyways and a major version bump ensures there is no
> > >> surprise.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jiangjie (Becket) Qin
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> > >>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Public API is a well defined common concept, and one of
> > >> its
> > >>>>>>>>>>>>> convention is that it only changes with a major version
> > >>>>> change.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public
> > >> API is
> > >>>>>> also a
> > >>>>>>>>> kind
> > >>>>>>>>>>>> of such change, just like removing one, which can only
> > >> happen
> > >>>>>> with
> > >>>>>>>>> major
> > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> > >>>> APIs
> > >>>>>>>>> anytime,
> > >>>>>>>>>>> but
> > >>>>>>>>>>>> only in the case major version bumps happen before
> > >> reaching
> > >>>> the
> > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> > >> would
> > >>>> be a
> > >>>>>>> weaker
> > >>>>>>>>>>>> change compared to removing the API immediately upon
> > >> major
> > >>>>>> version
> > >>>>>>>>> bumps,
> > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> > >>>>>> migration
> > >>>>>>>>>>> period.
> > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> > >> sounds
> > >>>>>>> against
> > >>>>>>>>>>>> conventions, we may introduce a new annotation if
> > >> necessary,
> > >>>>>> e.g.,
> > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> But it should be
> > >>>>>>>>>>>>> completely OK to bump up the major version if we really
> > >> want
> > >>>>> to
> > >>>>>>> get
> > >>>>>>>>> rid
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>> a public API, right?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> > >> that we
> > >>>>>> bump
> > >>>>>>> up
> > >>>>>>>>> the
> > >>>>>>>>>>>> major version whenever a breaking change is needed.
> > >> However,
> > >>>>> this
> > >>>>>>> also
> > >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> > >>>> stay
> > >>>>>>> stable
> > >>>>>>>>>>>> within the major release series, as the series can end
> > >>>> anytime.
> > >>>>>>> IMHO,
> > >>>>>>>>>>> short
> > >>>>>>>>>>>> major release series are not something "make the end
> > >> users
> > >>>>>> happy",
> > >>>>>>> but
> > >>>>>>>>>>>> backdoors that allow us as the developers to make
> > >> frequent
> > >>>>>> breaking
> > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> > >> can
> > >>>>>> still
> > >>>>>>>>> have
> > >>>>>>>>>>>> longer major release series, while only allowing Public
> > >> APIs
> > >>>>>>>>> deprecated
> > >>>>>>>>>>> at
> > >>>>>>>>>>>> the end of the previous major version to be removed in
> > >> the
> > >>>> next
> > >>>>>>> major
> > >>>>>>>>>>>> version.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > >> (1-2
> > >>>>> years)
> > >>>>>>> to
> > >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> > >>>> again
> > >>>>>> to
> > >>>>>>> the
> > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> > >>>> when
> > >>>>>>>>> breaking
> > >>>>>>>>>>>>> changes can be made.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
> > >> for new
> > >>>>>> major
> > >>>>>>>>>>>> versions, or longer if there's no breaking changes
> > >> needed.
> > >>>> That
> > >>>>>>> makes
> > >>>>>>>>> 1-2
> > >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving
> > >> us
> > >>>> more
> > >>>>>>> time
> > >>>>>>>>> to
> > >>>>>>>>>>>> prepare the major release if needed.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Xintong
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > >>>>>>> chesnay@apache.org <ma...@apache.org>
> > >>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
> > >>>>> versions
> > >>>>>>> for
> > >>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>> major versions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> > >> viable
> > >>>> that
> > >>>>>> is
> > >>>>>>> in
> > >>>>>>>>>>>>> practice.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> > >> support
> > >>>> for
> > >>>>>> Java
> > >>>>>>>>> 8/11,
> > >>>>>>>>>>>>> which may very well be a problem for our current users.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
> > >> about
> > >>>>>> Public
> > >>>>>>>>> API
> > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
> > >> in
> > >>>> the
> > >>>>>>> NOTES
> > >>>>>>>>> part
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
> > >> it.
> > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
> > >> are on
> > >>>>> the
> > >>>>>>> same
> > >>>>>>>>>>>>> page on that note.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > >>>>>>>>>>>>>> But it should be
> > >>>>>>>>>>>>>> completely OK to bump up the major version if we
> > >> really
> > >>>> want
> > >>>>>> to
> > >>>>>>> get
> > >>>>>>>>>>> rid
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>> a public API, right?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Technically yes, but look at how long it took to get us
> > >> to
> > >>>>> 2.0.
> > >>>>>> ;)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> There's a separate discussion to be had on the cadence
> > >> of
> > >>>>> major
> > >>>>>>>>> releases
> > >>>>>>>>>>>>> going forward, and there seem to be different opinions
> > >> on
> > >>>>> that.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between
> > >>>> major
> > >>>>>>> ones,
> > >>>>>>>>>>>>> that for us means that users have to potentially deal
> > >> with
> > >>>>>>> breaking
> > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle
> > >> (1-2
> > >>>>>> years)
> > >>>>>>> to
> > >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> > >>>> again
> > >>>>>> to
> > >>>>>>> the
> > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> > >>>> when
> > >>>>>>>>> breaking
> > >>>>>>>>>>>>> changes can be made.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> But again, maybe this should be in a separate thread.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
> > >> Not
> > >>>>> sure
> > >>>>>> I
> > >>>>>>>>> fully
> > >>>>>>>>>>>>>> understand the concern for behavioral change here.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This could be a lot of things. It can be performance in
> > >>>>> certain
> > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
> > >> relied
> > >>>>> upon
> > >>>>>>>>>>>>> (
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> ),
> > a semantic change to some
> > >> API.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> For a concrete example, consider the job submission. A
> > >> few
> > >>>>>>> releases
> > >>>>>>>>> back
> > >>>>>>>>>>>>> we made changes such that the initialization of the job
> > >>>> master
> > >>>>>>>>> happens
> > >>>>>>>>>>>>> asynchronously.
> > >>>>>>>>>>>>> This meant the job submission call returns sooner, and
> > >> the
> > >>>> job
> > >>>>>>> state
> > >>>>>>>>>>>>> enum was extended to cover this state.
> > >>>>>>>>>>>>> API-wise we consider this a compatible change, but the
> > >>>>> observed
> > >>>>>>>>> behavior
> > >>>>>>>>>>>>> may be different.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Metrics are another example; I believe over time we
> > >> changed
> > >>>>> what
> > >>>>>>> some
> > >>>>>>>>>>>>> metrics returned a few times.
> >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi all,

Thanks Xingtong for the summary. If I could only choose one of the given
two options, I would go with option 1. I understood that option 2 worked
great with Kafka. But the bridge release will still confuse users and my
gut feeling is that many users will skip 2.0 and be waiting for 3.0 or even
3.x. And since fewer users will use Flink 2.x, the development focus will
be on Flink 3.0 with the fact that the current Flink release is 1.17 and we
are preparing 2.0 release. That is weird for me.

THB, I would not name the change from @Public to @Retired as a demotion.
The purpose of @Retire is to extend the API lifecycle with one more stage,
like in the real world, people born, studied, graduated, worked, and
retired. Afaiu from the previous discussion, there are two rules we'd like
to follow simultaneously:

1. Public APIs can only be changed between major releases.
2. A smooth migration phase should be offered to users, i.e. at least 2
minor releases after APIs are marked as @deprecated. There should be new
APIs as the replacement.

Agree, those rules are good to improve the user friendliness. Issues we
discussed are rising because we want to fulfill both of them. If we take
care of deprecation very seriously, APIs can be marked as @Deprecated, only
when the new APIs as the replacement provide all functionalities the
deprecated APIs have. In an ideal case without critical bugs that might
stop users adopting the new APIs. Otherwise the expected "replacement" will
not happen. Users will still stick to the deprecated APIs, because the new
APIs can not be used. For big features, it will need at least 4 minor
releases(ideal case), i.e. 2+ years to remove deprecated APIs:

- 1st minor release to build the new APIs as the replacement and waiting
for feedback. It might be difficult to mark the old API as deprecated in
this release, because we are not sure if the new APIs could cover 100%
functionalities.
-  In the lucky case,  mark all old APIs as deprecated in the 2nd minor
release. (I would even suggest having the new APIs released at least for
two minor releases before marking it as deprecated to make sure they can
really replace the old APIs, in case we care more about smooth migration)
- 3rd minor release for the migration period
-  In another lucky case, the 4th release is a major release, the
deprecated APIs could be removed.

The above described scenario works only in an ideal case. In reality, it
might take longer to get the new APIs ready and mark the old API
deprecated. Furthermore, if the 4th release is not a major release, we will
have to maintain both APIs for many further minor releases. The question is
how to know the next major release in advance, especially 4 minor releases'
period, i.e. more than 2 years in advance? Given that Flink contains many
modules, it is difficult to ask devs to create a 2-3 years deprecation plan
for each case. In case we want to build major releases at a fast pace,
let's say every two years, it means devs must plan any API deprecation
right after each major release. Afaiac, it is quite difficult.

The major issue is, afaiu, if we follow rule 2, we have to keep all @Public
APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0. Then
we have to follow rule 1 to keep it unchanged until we have 3.0. That is
why @Retired is useful to give devs more flexibility and still fulfill both
rules. Let's check it with examples:

- we have @Public DataStream API in 1.18. It will not be marked
as @Deprecated, because the new APIs as the replacement are not ready.
- we keep the DataStream API itself unchanged in 2.0, but change the
annotation from @Public to @Retire. New APIs will be introduced too. In
this case, Rule 1 is ok, since public API is allowed to change between
major releases. Only changing annotation is the minimal change we could do
and it does not break rule 1. Rule 2 is ok too, since the DataStream APIs
work exactly the same.  Attention: the change of @Public -> @Retired can
only be done between major releases, because @Public APIs can only be
changed between major releases.
- in 2.1, DataStream API will be marked as deprecated.
- in 2.2, DataStream will be kept for the migration period.
- in 2.3, DataStream will be removed.

Becket mentioned previously (please correct me if I didn't understand it
correctly) that users might not check the changes of annotation and the
upgrade process might not be smooth. I was wondering, if users don't pay
attention to that, they will also not pay attention to @deprecated. The
migration will not be smooth too even when we stick to the Java
standard @deprecated. Let's go throw it with examples:

Scenario 1: with @Retired

Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be done
for users who still want to use DataStream. The migration is smooth. But
they should be aware of the annotation changes, i.e. @Public -> @Retired
Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since it is
a major release upgrade, breaking changes are expected.
Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed. Since APIs
are marked as @Retired, breaking changes are expected too.

In addition to fulfilling both rules, there are two more benefits of this
option:
1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
2. no bridge release is required. DataStream API could be deprecated in 2.x
without breaking any of the two rules. @Retired provides enough flexibility
for future deprecations and upgrades.

The only issue I can find is that users might be surprised in case 3, if
they ignore the previous @Public -> @Retired change. But for such users,
they will ignore @deprecated annotation too and will have issues with
pure @deprecated
annotation too:

Scenario 2: with @deprecated

The example will be slightly different from scenario 1, because @Public
APIs can only be removed between major releases. As an example, DataStream
API will be marked as deprecated in 1.19, kept in 1.20, and then removed in
2.0

Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as
deprecated are removed between major releases. Rule 1 is ok. The 2 minor
releases period defined in rule 2 sounds like a smooth migration plan, but
for users in this case, it is still not smooth.
Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with
deprecated APIs. But for those users who ignore annotation changes, will
have the same issue mentioned in case 1.

I understood that, compared to scenario 1, users, who ignore @Public
-> @Retired
changes, will not face any breaking changes, after they migrated to 2.0.
But the cost of it is bridge releases. There might be many bridge releases
in the future. As I mentioned previously, it is not easy to find the right
timing to kick off the deprecation process of APIs before the next expected
major release and 2-3 years in advance.

Scenario 3: @PublicEvolving deprecation for users who ignore @deprecated
annotation changes

Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21

Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth
migration.
Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
since @deprecated is ignored

In summary, for users who care about the annotation change, they will get
more benefits with @Retired. For users who ignore annotation changes, they
will be always facing non-smooth upgrades even using pure standard @deprecated
annotation/process.

For the case that IDEs support @deprecated, that is true. There are no such
supports for homebuilt annotation. But the intention is to let users be
aware of it. There will be other ways to do it for users who care about
annotation changes. For users who do not care, there should be no
difference between with tool support or without. Using @Retired with
@deprecation
together is at least not worse than only using @deprecation alone. And
there are additional benefits with @Retired.

Just my 2 cents and looking forward to your feedback, especially different
opinions that will help me understand the issue better. Thanks!


Best regards,
Jing






On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
<sr...@confluent.io.invalid> wrote:

> Hi Xintong,
>
> Thanks for the summary, most of the points that you agreed upon also make
> sense to me.
>
> >   2. Dropping deprecated Public APIs in minor releases, or demoting APIs
> >   from Public to PublicEvolving / Experimental / Retired in any version
> bump,
> >   are not good practices.
>
> I hope we can can go beyond calling it “not a good practice" and reach
> consensus that we will not demote or remove public APIs in minor releases
> and (technically) enforce this rule for all contributions. I think it’s a
> reasonable expectation for stability from a project as mature as Flink.
>
> > I'm personally in favor of option 1. As the migration-period rule is
> newly
> > proposed, I think it's fair to make exceptions for cases where we already
> > missed the best chance for planning the deprecation. Moreover, I do
> believe
> > having a quick major version bump does confuse users. Yes, we can explain
> > to users that bumping from 2.x to 3.0 does not cost anything other than
> the
> > removal of an deprecated API. But having to explain this itself is an
> > indicator that it might be confusing for users.
> >
> >
> > Becket, on the other hand, prefers option 2. From my understanding, his
> > major point is that a quick major version bump causes barely any actual
> > lose on users, while in option 1 not providing the migration period or
> > providing a shorter on is an actual lose on users. (@Becket, please
> correct
> > me if I'm mistaken.)
> >
>
> I would be open to both options and maybe postpone this decision until we
> have the complete roadmap for Flink 2.0. We would have a better overview
> about the technical consequences, e.g. how hard it would be to offer and
> maintain both APIs side by side, how well we get both APIs separated, and
> whether or not we will be able to use the full potential of our breaking
> changes before the DataStream API is completely removed. One example for
> the last point I can think of would be moving from lazy to eager state
> declaration. If we believe that it is reasonable to support all planned
> changes while maintaining both APIs, I'm leaning towards option 2.
>
> Best,
> Stefan
>
>
> > On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com> wrote:
> >
> > Becket and I had an offline voice call earlier today. We have reached
> > consensus on some of the arguments, but still hold different opinions on
> > some others. I'd like to post the outcome here for transparency.
> >
> > We both agree that:
> >
> >   1. Providing a migration period would be beneficial for our users, and
> >   we should do that
> >   2. Dropping deprecated Public APIs in minor releases, or demoting APIs
> >   from Public to PublicEvolving / Experimental / Retired in any version
> bump,
> >   are not good practices.
> >   3. Ideally, with this FLIP, developers should be more careful and plan
> >   API changes ahead. That means:
> >      1. Be more careful with designing APIs and promoting them to Public,
> >      so that they won't be changed / removed very soon, and also the
> >      maintenance overhead for keeping them after deprecation should be
> >      affordable. (I believe this also aligns with John's opinion.)
> >      2. Plan the deprecation / removal of old APIs that we don't want to
> >      carry to the next major version earlier, so that they'll have the
> >      2-minor-release migration period before the next major version bump.
> >   4. A practical situation is that, we are not ready for deprecating the
> >   DataStream API, nor afford to carry it for another couple of years.
> >      1. We cannot deprecate it now, because the new ProcessFunction API
> >      (the planned replacement for DataStream API) is not yet ready. We
> haven't
> >      planned the ProcessFunction API earlier because the original
> > plan was to do
> >      in-place changes directly on the DataStream API in release 2.0,
> until the
> >      smooth migration period is proposed. Before this, the only
> guarantees we
> >      provide for Public APIs is that they'll stay compatible until the
> next
> >      major version, which led to the understanding the in-place changes
> on
> >      Public APIs should be fine.
> >      2. We also cannot afford carrying the DataStream API for another
> >      couple of years. One of the reasons that we want to refactor /
> replace
> >      DataStream API is that it exposes and depends on too many
> > Flink's internal
> >      runtime implementations, which has already limited / blocked plans
> for
> >      Flink's internal improvements for many times. Also due to the wide
> and
> >      transitive dependencies on the internals, maintaining two sets of
> lower
> >      APIs itself is expensive and would be better to keep it short. This
> is
> >      admittedly a historical design flaw which should be avoided in
> future.
> >
> > What we haven't reached consensus is about how to deal with the current
> > situation around the DataStream API. There are two options.
> >
> >
> >   1. We say this FLIP is enforced starting release 2.0. For current 1.x
> >   APIs, we provide a migration period with best effort, while allowing
> >   exceptions for immediate removal in 2.0. That means we will still try
> with
> >   best effort to get the ProcessFuncion API ready and deprecate the
> >   DataStream API in 1.x, but will also be allowed to remove DataStream
> API in
> >   2.0 if it's not deprecated 2 minor releases before the major version
> bump.
> >   2. We strictly follow the process in this FLIP, and will quickly bump
> >   the major version from 2.x to 3.0 once the migration period for
> DataStream
> >   API is reached.
> >
> > I'm personally in favor of option 1. As the migration-period rule is
> newly
> > proposed, I think it's fair to make exceptions for cases where we already
> > missed the best chance for planning the deprecation. Moreover, I do
> believe
> > having a quick major version bump does confuse users. Yes, we can explain
> > to users that bumping from 2.x to 3.0 does not cost anything other than
> the
> > removal of an deprecated API. But having to explain this itself is an
> > indicator that it might be confusing for users.
> >
> >
> > Becket, on the other hand, prefers option 2. From my understanding, his
> > major point is that a quick major version bump causes barely any actual
> > lose on users, while in option 1 not providing the migration period or
> > providing a shorter on is an actual lose on users. (@Becket, please
> correct
> > me if I'm mistaken.)
> >
> >
> > And we'd like to hear more feedback from the community.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcephei@apache.org
> <ma...@apache.org>> wrote:
> >
> >> Hi Becket and Xintong,
> >>
> >> I hope you don't mind if I chime in a little.
> >>
> >> Once an API is marked Public, we're committing to support it until it's
> >> deprecated, and once it's deprecated, to leave it in place for at least
> two
> >> minor releases and then only remove it in the following major release.
> >>
> >> As a user, I would be dismayed to discover that the project maintainers
> >> demoted a Public API to PublicEvolving or Experimental in order to
> violate
> >> the spirit of the deprecation period for Public APIs. Likewise, I'd
> view a
> >> rapid sequence of minor releases for the purpose of dropping deprecated
> >> APIs in the next major release as another violation of the spirit of our
> >> guarantees. I'm reminded of this xkcd:
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> >>
> >> I'm glad we're gaming out these situations before we institute this
> FLIP,
> >> and I hope we all converge on a clear understanding of what we're
> >> guaranteeing. From where I'm sitting, the essence of this FLIP is
> >> specifically to prevent us from doing whatever we want and instead to
> >> commit to providing a suitable notice period to users who rely on Public
> >> APIs.
> >>
> >> It honestly shouldn't matter what the maintenance overhead is (or,
> rather,
> >> evaluating the maintenance overhead should be a consideration before we
> >> promote an API to Public to begin with). My experience in other
> projects is
> >> that regretting a feature so much you want to drop it asap is extremely
> >> rare, and that living with the pain for a little while until the
> >> deprecation period expires will probably make us better engineers.
> >>
> >> With that all said, one thing that is not a violation is to propose a
> new
> >> major release in order to drop especially burdensome deprecated APIs
> that
> >> have already enjoyed their full deprecation period. Those really bad
> "case
> >> A" features will be very rare.
> >>
> >> Looking over the FLIP again, I see a reference to the release
> >> documentation, but not to the build tooling support for the Deprecated
> >> annotation, so it might help to share this scenario: The way this will
> all
> >> come together in practice is that, by giving good stability guarantees,
> we
> >> will encourage our users to bump minor versions when they are released,
> >> which means that they will see compiler warnings immediately if any of
> the
> >> APIs they've used are deprecated. In fact, I'd recommend for them to use
> >> the `-Werror` javac flag to be sure that they don't miss those warnings.
> >> Given our release cadence, this means that they will have about six
> months
> >> to migrate off of any deprecated APIs without being forced to avoid
> >> updates. And they really need to be able to adopt further minor updates
> in
> >> the series, since they may contain further improvements that actually
> >> facilitate migrating off of the deprecated APIs.
> >>
> >> The compiler warnings are a far more reliable mechanism to advertise
> >> deprecation than release notes, since users' own builds will notify them
> >> right away about only the deprecations that are relevant to them, but it
> >> does rely on users feeling ok to bump up minor versions in a timely
> >> fashion. If we consistently burn them by not observing the stability
> >> guarantees, they'll try to avoid updates instead, and the whole thing
> falls
> >> apart.
> >>
> >> Thanks again,
> >> -John
> >>
> >> On 2023/06/19 10:43:05 Becket Qin wrote:
> >>> Hi Xintong,
> >>>
> >>> Let's compare the following cases:
> >>>
> >>> A. If the maintenance overhead of the deprecated API is high, and we
> want
> >>> to remove it after two minor releases. Then there are two options:
> >>>    A1: Demote the API in the major version bump and remove the code
> >> with a
> >>> minor version bump.
> >>>    A2: Do exactly the same as 1-A, except that when removing the code,
> >>> bump up the major version. So this might be a short major version.
> >>>
> >>> B. If the maintenance overhead of the deprecated API is not high,
> >> therefore
> >>> keeping it for a long time is affordable. There are also two options:
> >>>    B1: Same as A-1, demote the API in the major version bump and remove
> >>> the code with a minor version bump.
> >>>    B2: Keep the API for all the minor versions in the major version,
> and
> >>> only remove the code in the next major version.
> >>>
> >>> For case B, do we agree that B2 is the way to go?
> >>>
> >>> For case A:
> >>> The following stays the same for A1 and A2:
> >>>  - users will lose the API after two minor leases. So the migration
> >> period
> >>> is the same for A-1 and A-2.
> >>>  - A1 and A2 will have exactly the same code after the removal of
> >>> deprecated API, the only difference is versioning.
> >>>  - To move forward, users need to move to the next minor version in A1,
> >> or
> >>> to the next major version in A2. Because the code are the same, the
> >> actual
> >>> effort to upgrade is the same for A1 and A2.
> >>>
> >>> The differences between A-1 and A-2 are:
> >>>  - A1 allows keeping the major version release cadence. A-2 will have a
> >>> short major version release.
> >>>  - A1 breaks the well understood API semantic, while A-2 does not.
> >>>
> >>> From what I see, since there is no well establish standard regarding
> how
> >>> long a major version should be released, A short major version release
> is
> >>> potential and emotional. It is not ideal but does not have material
> >>> downsides compared with A1. I did not hear anyone complaining about
> Kafka
> >>> only has two 1.x release. However, A1 actually breaks the well
> understood
> >>> API semantic, which has more material impact.
> >>>
> >>> Also, I'd imagine 90% or more of the Public APIs should fall into case
> B.
> >>> So, short major versions should be very occasional. I'd be very
> concerned
> >>> if the reason we choose A1 is simply because we cannot afford
> >> maintaining a
> >>> bunch of deprecated APIs until the next major version. This indicates
> >> that
> >>> the actual problem we need to solve is to lower the maintenance
> overhead
> >> of
> >>> deprecated APIs, so that we are comfortable to keep them longer. As
> John
> >>> and I mentioned earlier, there are ways to achieve this and we need to
> >>> learn how to do it in Flink. Otherwise, our discussion about versioning
> >>> here does not bring much value, because we will end up with a bunch of
> >>> short-lived APIs which upset our users, no matter how we version the
> >>> releases.
> >>>
> >>> So, if there are concrete examples that you think will block us from
> >>> keeping API stability with affordable cost, let's take a look together
> >> and
> >>> see if that can be improved.
> >>>
> >>> Thanks,
> >>>
> >>> Jiangjie (Becket) Qin
> >>>
> >>>
> >>> From what I see,
> >>>
> >>>
> >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong820@gmail.com
> <ma...@gmail.com>>
> >> wrote:
> >>>
> >>>>>
> >>>>> The part I don't understand is if we are willing to have a migration
> >>>>> period, and do a minor version bump to remove an API, what do we
> >> lose to
> >>>> do
> >>>>> a major version bump instead, so we don't break the common versioning
> >>>>> semantic?
> >>>>>
> >>>>
> >>>> I think we are talking about the cases where a major version bump
> >> happens
> >>>> before a deprecated Public API reaches its migration period. So
> >> removing
> >>>> the API with another major version bump means we have two consecutive
> >> major
> >>>> versions in a very short time. And as previously mentioned, having
> >> frequent
> >>>> major version bumps would weaken the value of the commitment "Public
> >> API
> >>>> stay compatible within a major version". I think users also have
> >>>> expectations about how long a major version should live, which should
> >> be at
> >>>> least a couple of years rather than 1-2 minor releases.
> >>>>
> >>>> This is another option, but I think it is very likely more expensive
> >> than
> >>>>> simply bumping the major version. And I imagine there will be
> >> questions
> >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> >>>>>
> >>>>
> >>>> Yes, it's more expensive than simply bumping the major version, but
> >> IMHO
> >>>> it's probably cheaper than carrying the API until the next major
> >> version if
> >>>> we don't bump the major version very soon. And see my reply above on
> >> why
> >>>> not bumping immediately.
> >>>>
> >>>> Best,
> >>>>
> >>>> Xintong
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket.qin@gmail.com
> <ma...@gmail.com>>
> >> wrote:
> >>>>
> >>>>> Hi Xintong,
> >>>>>
> >>>>> Please see the replies below.
> >>>>>
> >>>>> I see your point, that users would feel surprised if they find
> >> things no
> >>>>>> longer work when upgrading to another 2.x minor release. However,
> >> I'd
> >>>>> like
> >>>>>> to point out that PublicEvolving APIs would have the similar
> >> problem
> >>>>>> anyway. So the question is, how do we catch users' attention and
> >> make
> >>>>> sure
> >>>>>> they are aware that the Public APIs in 1.x may no longer be Public
> >> in
> >>>>> 2.0.
> >>>>>> There're various ways to do that, e.g., release notes, warnings in
> >>>> logs,
> >>>>>> etc.
> >>>>>
> >>>>> First of all, I am not a fan of removing PublicEvolving APIs in minor
> >>>>> version changes. Personally speaking, I want them to be also removed
> >> in
> >>>>> major version changes.
> >>>>>
> >>>>> Empirically, I rarely see projects with complex API semantic work
> >> well.
> >>>>> Many, if not most, users don't read docs / release notes / warnings
> >> in
> >>>>> logs. I don't think that is their fault. Typically an application
> >>>> developer
> >>>>> will have to deal with dozens of libraries maintained by various
> >>>>> communities / groups. It is just too difficult for them to keep
> >> track of
> >>>>> all the specific semantics each project puts there. If you think
> >> about
> >>>> it,
> >>>>> how many of Flink developers read all the release notes of Guava,
> >> Apache
> >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of
> >> them?
> >>>>> One would probably try bumping the dependency version and see if
> >> there is
> >>>>> an exception and solve them case by case. And if it is a minor
> >> version
> >>>>> bump, one probably does not expect an exception at all. Another
> >> example
> >>>> is
> >>>>> that in Flink we still have so many usage of deprecated methods all
> >> over
> >>>>> the place, and I strongly doubt everyone knows when the source code
> >> of
> >>>>> these methods are deprecated and when they should be removed.
> >>>>>
> >>>>> So, the most important principle of API is simple and intuitive. The
> >>>>> versioning semantic is a simple and universally accepted API
> >> stability
> >>>>> standard. If we ourselves as Flink developers are relying on this
> >> for our
> >>>>> own dependencies. I don't think we can expect more from our users.
> >>>>>
> >>>>> Another possible alternative: whenever there's a deprecated Public
> >> API
> >>>> that
> >>>>>> reaches a major version bump before the migration period, and we
> >> also
> >>>>> don't
> >>>>>> want to carry it for all the next major release series, we may
> >> consider
> >>>>>> releasing more minor releases for the previous major version after
> >> the
> >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
> >> to
> >>>> 2.0,
> >>>>>> we can release one more 1.20 after 2.0. That should provide users
> >>>> another
> >>>>>> choice rather than upgrading to 2.0, while satisfying the
> >>>> 2-minor-release
> >>>>>> migration period.
> >>>>>
> >>>>> This is another option, but I think it is very likely more expensive
> >> than
> >>>>> simply bumping the major version. And I imagine there will be
> >> questions
> >>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
> >>>>>
> >>>>> I think my major point is, we should not carry APIs deprecated in a
> >>>>>> previous major version along all the next major version series. I'd
> >>>> like
> >>>>> to
> >>>>>> try giving users more commitments, i.e. the migration period, as
> >> long
> >>>> as
> >>>>> it
> >>>>>> does not prevent us from making breaking changes. If it doesn't
> >> work,
> >>>> I'd
> >>>>>> be in favor of not providing the migration period, but fallback to
> >> only
> >>>>>> guarantee the compatibility within the major version.
> >>>>>
> >>>>> The part I don't understand is if we are willing to have a migration
> >>>>> period, and do a minor version bump to remove an API, what do we
> >> lose to
> >>>> do
> >>>>> a major version bump instead, so we don't break the common versioning
> >>>>> semantic?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jiangjie (Becket) Qin
> >>>>>
> >>>>>
> >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <tonysong820@gmail.com
> <ma...@gmail.com>>
> >>>>> wrote:
> >>>>>
> >>>>>>>
> >>>>>>> As an end user who only uses Public APIs, if I don't change my
> >> code
> >>>> at
> >>>>>>> all, my expectation is the following:
> >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> >> with
> >>>>> all
> >>>>>>> the 2.x versions.
> >>>>>>> Actually I think there are some dependency version resolution
> >>>> policies
> >>>>>> out
> >>>>>>> there which picks the highest minor version when the dependencies
> >>>> pull
> >>>>> in
> >>>>>>> multiple minor versions of the same jar, which may be broken if
> >> we
> >>>>> remove
> >>>>>>> the API in minor releases.
> >>>>>>>
> >>>>>>
> >>>>>> I see your point, that users would feel surprised if they find
> >> things
> >>>> no
> >>>>>> longer work when upgrading to another 2.x minor release. However,
> >> I'd
> >>>>> like
> >>>>>> to point out that PublicEvolving APIs would have the similar
> >> problem
> >>>>>> anyway. So the question is, how do we catch users' attention and
> >> make
> >>>>> sure
> >>>>>> they are aware that the Public APIs in 1.x may no longer be Public
> >> in
> >>>>> 2.0.
> >>>>>> There're various ways to do that, e.g., release notes, warnings in
> >>>> logs,
> >>>>>> etc.
> >>>>>>
> >>>>>> Another possible alternative: whenever there's a deprecated Public
> >> API
> >>>>> that
> >>>>>> reaches a major version bump before the migration period, and we
> >> also
> >>>>> don't
> >>>>>> want to carry it for all the next major release series, we may
> >> consider
> >>>>>> releasing more minor releases for the previous major version after
> >> the
> >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
> >> to
> >>>> 2.0,
> >>>>>> we can release one more 1.20 after 2.0. That should provide users
> >>>> another
> >>>>>> choice rather than upgrading to 2.0, while satisfying the
> >>>> 2-minor-release
> >>>>>> migration period.
> >>>>>>
> >>>>>> I think my major point is, we should not carry APIs deprecated in a
> >>>>>> previous major version along all the next major version series. I'd
> >>>> like
> >>>>> to
> >>>>>> try giving users more commitments, i.e. the migration period, as
> >> long
> >>>> as
> >>>>> it
> >>>>>> does not prevent us from making breaking changes. If it doesn't
> >> work,
> >>>> I'd
> >>>>>> be in favor of not providing the migration period, but fallback to
> >> only
> >>>>>> guarantee the compatibility within the major version.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Xintong
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcephei@apache.org
> <ma...@apache.org>
> >>>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi Becket,
> >>>>>>>
> >>>>>>> Thanks for the reply! I’d like to continue the conversation about
> >>>>>>> compatibility outside of this FLIP thread, but for now, I can
> >> accept
> >>>>> your
> >>>>>>> decision. It’s certainly an improvement.
> >>>>>>>
> >>>>>>> Thanks again,
> >>>>>>> John
> >>>>>>>
> >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> >>>>>>>> Hi John,
> >>>>>>>>
> >>>>>>>> Completely agree with all you said.
> >>>>>>>>
> >>>>>>>> Can we consider only dropping deprecated APIs in major releases
> >>>>> across
> >>>>>>> the
> >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> >> are
> >>>> by
> >>>>>>>>> definition less stable, but it seems like this should be
> >> reflected
> >>>>> in
> >>>>>>> the
> >>>>>>>>> required deprecation period alone. I.e. that we must keep them
> >>>>> around
> >>>>>>> for
> >>>>>>>>> at least zero or one minor release, not that we can drop them
> >> in a
> >>>>>>> minor or
> >>>>>>>>> patch release.
> >>>>>>>>
> >>>>>>>> Personally speaking, I would love to do this, for exactly the
> >>>> reason
> >>>>>> you
> >>>>>>>> mentioned. However, I did not propose this due to the following
> >>>>>> reasons:
> >>>>>>>>
> >>>>>>>> 1. I am hesitating a little bit about changing the accepted
> >> FLIPs
> >>>> too
> >>>>>>> soon.
> >>>>>>>> 2. More importantly, to avoid slowing down our development. At
> >> this
> >>>>>>> point,
> >>>>>>>> Flink still lacks some design / routines to support good API
> >>>>>>> evolvability /
> >>>>>>>> extensibility. Just like you said, it takes some time to be
> >> good at
> >>>>>> this.
> >>>>>>>> In this case, my concern is that only removing Experimental /
> >>>>>>>> PublicEvolving APIs in major version changes may result in too
> >> much
> >>>>>>>> overhead and dramatically slow down the development of Flink.
> >> So, I
> >>>>> was
> >>>>>>>> thinking that we can start with the current status. Hopefully
> >> after
> >>>>> we
> >>>>>>> are
> >>>>>>>> more comfortable with the maintenance overhead of deprecated
> >> APIs,
> >>>> we
> >>>>>> can
> >>>>>>>> then have a stronger guarantee for Experimental /
> >> PublicEvolving
> >>>>> APIs.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jiangjie (Becket) Qin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> >> vvcephei@apache.org <ma...@apache.org>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Becket,
> >>>>>>>>>
> >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really
> >>>>>> important.
> >>>>>>> I
> >>>>>>>>> understand some people’s concerns about the additional burden
> >> for
> >>>>>>> project
> >>>>>>>>> maintainers, but my personal experience with Kafka has been
> >> that
> >>>>> it’s
> >>>>>>> very
> >>>>>>>>> liveable and that it’s well worth the benefit to users. In
> >> fact,
> >>>>> users
> >>>>>>>>> being able to confidently upgrade is also a benefit to
> >>>> maintainers,
> >>>>> as
> >>>>>>> we
> >>>>>>>>> will get fewer questions from people stuck on very old
> >> versions.
> >>>>>>>>>
> >>>>>>>>> One question:
> >>>>>>>>> Can we consider only dropping deprecated APIs in major
> >> releases
> >>>>> across
> >>>>>>> the
> >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
> >> are
> >>>> by
> >>>>>>>>> definition less stable, but it seems like this should be
> >> reflected
> >>>>> in
> >>>>>>> the
> >>>>>>>>> required deprecation period alone. I.e. that we must keep them
> >>>>> around
> >>>>>>> for
> >>>>>>>>> at least zero or one minor release, not that we can drop them
> >> in a
> >>>>>>> minor or
> >>>>>>>>> patch release.
> >>>>>>>>>
> >>>>>>>>> The advantage of forbidding the removal of any API in minor or
> >>>> patch
> >>>>>>>>> releases is that users will get a strong guarantee that they
> >> can
> >>>>> bump
> >>>>>>> the
> >>>>>>>>> minor or patch version and still be able to compile, or even
> >> just
> >>>>>>> re-link
> >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run
> >> time.
> >>>>> This
> >>>>>>> is a
> >>>>>>>>> binary guarantee: if we allow removing  even Experimental APIs
> >>>>> outside
> >>>>>>> of
> >>>>>>>>> major releases, users can no longer confidently upgrade.
> >>>>>>>>>
> >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points:
> >>>>>>>>> * I’d use the official Deprecated annotation instead of
> >>>> introducing
> >>>>>> our
> >>>>>>>>> own flavor (Retired, etc), since Deprecated is well integrated
> >>>> into
> >>>>>>> build
> >>>>>>>>> tools and IDEs.
> >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it
> >> seems
> >>>>>>>>> orthogonal, and something that should probably be taken
> >>>> case-by-case
> >>>>>>>>> anyway.
> >>>>>>>>> * Aside from deprecation and removal, there have been some
> >>>>> discussions
> >>>>>>>>> about how to evolve APIs and behavior in compatible ways.
> >> This is
> >>>>>>> somewhat
> >>>>>>>>> of an art, and if folks haven’t wrestled with it before, it’ll
> >>>> take
> >>>>>> some
> >>>>>>>>> time to become good at it. I feel like this topic should also
> >> be
> >>>>>>> orthogonal
> >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
> >> simple
> >>>>>> policy
> >>>>>>> not
> >>>>>>>>> to break existing user programs, and leave the “how” up to
> >>>>>> implementers
> >>>>>>> and
> >>>>>>>>> reviewers.
> >>>>>>>>>
> >>>>>>>>> Thanks again,
> >>>>>>>>> John
> >>>>>>>>>
> >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> >>>>>>>>>> Hi All,
> >>>>>>>>>>
> >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
> >> great
> >>>>> idea.
> >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> >>>> @PublicEvolving
> >>>>>> --(2
> >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
> >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
> >> flexibility
> >>>>>>> without
> >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
> >> change
> >>>>>> between
> >>>>>>>>> major
> >>>>>>>>>> releases. Changing annotations is acceptable and provides
> >>>>> additional
> >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself
> >> are
> >>>> not
> >>>>>>>>> changed.
> >>>>>>>>>>
> >>>>>>>>>> I had similar thoughts when I was facing those issues. I
> >> want to
> >>>>>> move
> >>>>>>> one
> >>>>>>>>>> step further and suggest introducing one more annotation
> >>>> @Retired.
> >>>>>>>>>>
> >>>>>>>>>> Not like the @PublicRetired which is a compromise of
> >> downgrading
> >>>>>>> @Public
> >>>>>>>>> to
> >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java
> >>>> standard
> >>>>>>>>>> @deprecated should be used in the early stage of the
> >> deprecation
> >>>>>>> process
> >>>>>>>>>> and doesn't really meet our requirement. Since Java does not
> >>>> allow
> >>>>>> us
> >>>>>>> to
> >>>>>>>>>> extend annotation, I think it would be feasible to have the
> >> new
> >>>>>>> @Retired
> >>>>>>>>> to
> >>>>>>>>>> help us monitor and manage the deprecation process, house
> >>>>> cleaning,
> >>>>>>> etc.
> >>>>>>>>>>
> >>>>>>>>>> Some ideas could be(open for discussion):
> >>>>>>>>>>
> >>>>>>>>>> @Retired:
> >>>>>>>>>>
> >>>>>>>>>> 1. There must be a replacement with functionality
> >> compatibility
> >>>>>> before
> >>>>>>>>> APIs
> >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes
> >> on
> >>>> the
> >>>>> ML
> >>>>>>> are
> >>>>>>>>>> mandatory (a FLIP is recommended).
> >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
> >> release
> >>>>>>> sharply
> >>>>>>>>>> (using ArchUnit to force it, needs further check whether it
> >> is
> >>>>>>> possible).
> >>>>>>>>>> Devs who marked them as @Retired are responsible to remove
> >> them.
> >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
> >> are
> >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
> >> @Retired
> >>>>>> could
> >>>>>>>>> also
> >>>>>>>>>> be used if it can increase user-friendliness or
> >>>> dev-friendliness,
> >>>>>> but
> >>>>>>> not
> >>>>>>>>>> mandatory.
> >>>>>>>>>> 4. Some variables will be defined in @Retired to support the
> >>>>>>> deprecation
> >>>>>>>>>> process management. Further extension is possible, since the
> >>>>>>> annotation
> >>>>>>>>> is
> >>>>>>>>>> built by us.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best regards,
> >>>>>>>>>> Jing
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> >>>> becket.qin@gmail.com <ma...@gmail.com>
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Xintong,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the explanation. Please see the replies inline
> >>>> below.
> >>>>>>>>>>>
> >>>>>>>>>>> I agree. And from my understanding, demoting a Public API
> >> is
> >>>>> also a
> >>>>>>>>> kind of
> >>>>>>>>>>>> such change, just like removing one, which can only
> >> happen
> >>>> with
> >>>>>>> major
> >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> >>>> APIs
> >>>>>>>>> anytime,
> >>>>>>>>>>> but
> >>>>>>>>>>>> only in the case major version bumps happen before
> >> reaching
> >>>> the
> >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> >> would
> >>>> be a
> >>>>>>> weaker
> >>>>>>>>>>>> change compared to removing the API immediately upon
> >> major
> >>>>>> version
> >>>>>>>>> bumps,
> >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> >>>>>> migration
> >>>>>>>>>>> period.
> >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> >> sounds
> >>>>>>> against
> >>>>>>>>>>>> conventions, we may introduce a new annotation if
> >> necessary,
> >>>>>> e.g.,
> >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> >>>>>>>>>>>
> >>>>>>>>>>> As an end user who only uses Public APIs, if I don't
> >> change my
> >>>>> code
> >>>>>>> at
> >>>>>>>>> all,
> >>>>>>>>>>> my expectation is the following:
> >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> >> fine
> >>>>>> with
> >>>>>>> all
> >>>>>>>>>>> the 2.x versions.
> >>>>>>>>>>> Actually I think there are some dependency version
> >> resolution
> >>>>>>> policies
> >>>>>>>>> out
> >>>>>>>>>>> there which picks the highest minor version when the
> >>>> dependencies
> >>>>>>> pull
> >>>>>>>>> in
> >>>>>>>>>>> multiple minor versions of the same jar, which may be
> >> broken if
> >>>>> we
> >>>>>>>>> remove
> >>>>>>>>>>> the API in minor releases.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that
> >> we
> >>>>> bump
> >>>>>> up
> >>>>>>>>> the
> >>>>>>>>>>>> major version whenever a breaking change is needed.
> >> However,
> >>>>> this
> >>>>>>> also
> >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> >>>> stay
> >>>>>>> stable
> >>>>>>>>>>>> within the major release series, as the series can end
> >>>> anytime.
> >>>>>>> IMHO,
> >>>>>>>>>>> short
> >>>>>>>>>>>> major release series are not something "make the end
> >> users
> >>>>>> happy",
> >>>>>>> but
> >>>>>>>>>>>> backdoors that allow us as the developers to make
> >> frequent
> >>>>>> breaking
> >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> >> can
> >>>>>> still
> >>>>>>>>> have
> >>>>>>>>>>>> longer major release series, while only allowing Public
> >> APIs
> >>>>>>>>> deprecated
> >>>>>>>>>>> at
> >>>>>>>>>>>> the end of the previous major version to be removed in
> >> the
> >>>> next
> >>>>>>> major
> >>>>>>>>>>>> version.
> >>>>>>>>>>>
> >>>>>>>>>>> I totally agree that frequent major version bumps are not
> >>>> ideal,
> >>>>>> but
> >>>>>>>>> here
> >>>>>>>>>>> we are comparing it with a minor version bump which
> >> removes a
> >>>>>> Public
> >>>>>>>>> API.
> >>>>>>>>>>> So the context is that we have already decided to remove
> >> this
> >>>>>> Public
> >>>>>>> API
> >>>>>>>>>>> while keeping everything else backwards compatible. I
> >> think a
> >>>>> major
> >>>>>>>>> version
> >>>>>>>>>>> bump is a commonly understood signal here, compared with a
> >>>> minor
> >>>>>>> version
> >>>>>>>>>>> change. From end users' perspective, for those who are not
> >>>>>> impacted,
> >>>>>>> in
> >>>>>>>>>>> this case upgrading a major version is not necessarily more
> >>>>>> involved
> >>>>>>>>> than
> >>>>>>>>>>> upgrading a minor version - both should be as smooth as a
> >>>>>> dependency
> >>>>>>>>>>> version change. For those who are impacted, they will lose
> >> the
> >>>>>> Public
> >>>>>>>>> API
> >>>>>>>>>>> anyways and a major version bump ensures there is no
> >> surprise.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jiangjie (Becket) Qin
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> >>>>>> tonysong820@gmail.com <ma...@gmail.com>
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Public API is a well defined common concept, and one of
> >> its
> >>>>>>>>>>>>> convention is that it only changes with a major version
> >>>>> change.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I agree. And from my understanding, demoting a Public
> >> API is
> >>>>>> also a
> >>>>>>>>> kind
> >>>>>>>>>>>> of such change, just like removing one, which can only
> >> happen
> >>>>>> with
> >>>>>>>>> major
> >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
> >>>> APIs
> >>>>>>>>> anytime,
> >>>>>>>>>>> but
> >>>>>>>>>>>> only in the case major version bumps happen before
> >> reaching
> >>>> the
> >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
> >> would
> >>>> be a
> >>>>>>> weaker
> >>>>>>>>>>>> change compared to removing the API immediately upon
> >> major
> >>>>>> version
> >>>>>>>>> bumps,
> >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
> >>>>>> migration
> >>>>>>>>>>> period.
> >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
> >> sounds
> >>>>>>> against
> >>>>>>>>>>>> conventions, we may introduce a new annotation if
> >> necessary,
> >>>>>> e.g.,
> >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> But it should be
> >>>>>>>>>>>>> completely OK to bump up the major version if we really
> >> want
> >>>>> to
> >>>>>>> get
> >>>>>>>>> rid
> >>>>>>>>>>> of
> >>>>>>>>>>>>> a public API, right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
> >> that we
> >>>>>> bump
> >>>>>>> up
> >>>>>>>>> the
> >>>>>>>>>>>> major version whenever a breaking change is needed.
> >> However,
> >>>>> this
> >>>>>>> also
> >>>>>>>>>>>> weakens the value of the commitment that public APIs will
> >>>> stay
> >>>>>>> stable
> >>>>>>>>>>>> within the major release series, as the series can end
> >>>> anytime.
> >>>>>>> IMHO,
> >>>>>>>>>>> short
> >>>>>>>>>>>> major release series are not something "make the end
> >> users
> >>>>>> happy",
> >>>>>>> but
> >>>>>>>>>>>> backdoors that allow us as the developers to make
> >> frequent
> >>>>>> breaking
> >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
> >> can
> >>>>>> still
> >>>>>>>>> have
> >>>>>>>>>>>> longer major release series, while only allowing Public
> >> APIs
> >>>>>>>>> deprecated
> >>>>>>>>>>> at
> >>>>>>>>>>>> the end of the previous major version to be removed in
> >> the
> >>>> next
> >>>>>>> major
> >>>>>>>>>>>> version.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Given our track record I would prefer a regular cycle
> >> (1-2
> >>>>> years)
> >>>>>>> to
> >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> >>>> again
> >>>>>> to
> >>>>>>> the
> >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> >>>> when
> >>>>>>>>> breaking
> >>>>>>>>>>>>> changes can be made.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
> >> for new
> >>>>>> major
> >>>>>>>>>>>> versions, or longer if there's no breaking changes
> >> needed.
> >>>> That
> >>>>>>> makes
> >>>>>>>>> 1-2
> >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving
> >> us
> >>>> more
> >>>>>>> time
> >>>>>>>>> to
> >>>>>>>>>>>> prepare the major release if needed.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Xintong
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> >>>>>>> chesnay@apache.org <ma...@apache.org>
> >>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
> >>>>> versions
> >>>>>>> for
> >>>>>>>>>>>>> previous
> >>>>>>>>>>>>>> major versions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
> >> viable
> >>>> that
> >>>>>> is
> >>>>>>> in
> >>>>>>>>>>>>> practice.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
> >> support
> >>>> for
> >>>>>> Java
> >>>>>>>>> 8/11,
> >>>>>>>>>>>>> which may very well be a problem for our current users.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
> >> about
> >>>>>> Public
> >>>>>>>>> API
> >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
> >> in
> >>>> the
> >>>>>>> NOTES
> >>>>>>>>> part
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
> >> it.
> >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
> >> are on
> >>>>> the
> >>>>>>> same
> >>>>>>>>>>>>> page on that note.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> >>>>>>>>>>>>>> But it should be
> >>>>>>>>>>>>>> completely OK to bump up the major version if we
> >> really
> >>>> want
> >>>>>> to
> >>>>>>> get
> >>>>>>>>>>> rid
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>> a public API, right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Technically yes, but look at how long it took to get us
> >> to
> >>>>> 2.0.
> >>>>>> ;)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> There's a separate discussion to be had on the cadence
> >> of
> >>>>> major
> >>>>>>>>> releases
> >>>>>>>>>>>>> going forward, and there seem to be different opinions
> >> on
> >>>>> that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between
> >>>> major
> >>>>>>> ones,
> >>>>>>>>>>>>> that for us means that users have to potentially deal
> >> with
> >>>>>>> breaking
> >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Given our track record I would prefer a regular cycle
> >> (1-2
> >>>>>> years)
> >>>>>>> to
> >>>>>>>>>>>>> force us to think about this whole topic, and not put it
> >>>> again
> >>>>>> to
> >>>>>>> the
> >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
> >>>> when
> >>>>>>>>> breaking
> >>>>>>>>>>>>> changes can be made.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> But again, maybe this should be in a separate thread.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> >>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
> >> Not
> >>>>> sure
> >>>>>> I
> >>>>>>>>> fully
> >>>>>>>>>>>>>> understand the concern for behavioral change here.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This could be a lot of things. It can be performance in
> >>>>> certain
> >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
> >> relied
> >>>>> upon
> >>>>>>>>>>>>> (
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I),
> a semantic change to some
> >> API.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For a concrete example, consider the job submission. A
> >> few
> >>>>>>> releases
> >>>>>>>>> back
> >>>>>>>>>>>>> we made changes such that the initialization of the job
> >>>> master
> >>>>>>>>> happens
> >>>>>>>>>>>>> asynchronously.
> >>>>>>>>>>>>> This meant the job submission call returns sooner, and
> >> the
> >>>> job
> >>>>>>> state
> >>>>>>>>>>>>> enum was extended to cover this state.
> >>>>>>>>>>>>> API-wise we consider this a compatible change, but the
> >>>>> observed
> >>>>>>>>> behavior
> >>>>>>>>>>>>> may be different.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Metrics are another example; I believe over time we
> >> changed
> >>>>> what
> >>>>>>> some
> >>>>>>>>>>>>> metrics returned a few times.
>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Stefan Richter <sr...@confluent.io.INVALID>.
Hi Xintong,

Thanks for the summary, most of the points that you agreed upon also make sense to me.

>   2. Dropping deprecated Public APIs in minor releases, or demoting APIs
>   from Public to PublicEvolving / Experimental / Retired in any version bump,
>   are not good practices.

I hope we can can go beyond calling it “not a good practice" and reach consensus that we will not demote or remove public APIs in minor releases
and (technically) enforce this rule for all contributions. I think it’s a reasonable expectation for stability from a project as mature as Flink.

> I'm personally in favor of option 1. As the migration-period rule is newly
> proposed, I think it's fair to make exceptions for cases where we already
> missed the best chance for planning the deprecation. Moreover, I do believe
> having a quick major version bump does confuse users. Yes, we can explain
> to users that bumping from 2.x to 3.0 does not cost anything other than the
> removal of an deprecated API. But having to explain this itself is an
> indicator that it might be confusing for users.
> 
> 
> Becket, on the other hand, prefers option 2. From my understanding, his
> major point is that a quick major version bump causes barely any actual
> lose on users, while in option 1 not providing the migration period or
> providing a shorter on is an actual lose on users. (@Becket, please correct
> me if I'm mistaken.)
> 

I would be open to both options and maybe postpone this decision until we have the complete roadmap for Flink 2.0. We would have a better overview about the technical consequences, e.g. how hard it would be to offer and maintain both APIs side by side, how well we get both APIs separated, and whether or not we will be able to use the full potential of our breaking changes before the DataStream API is completely removed. One example for the last point I can think of would be moving from lazy to eager state declaration. If we believe that it is reasonable to support all planned changes while maintaining both APIs, I'm leaning towards option 2.

Best,
Stefan


> On 20. Jun 2023, at 14:43, Xintong Song <to...@gmail.com> wrote:
> 
> Becket and I had an offline voice call earlier today. We have reached
> consensus on some of the arguments, but still hold different opinions on
> some others. I'd like to post the outcome here for transparency.
> 
> We both agree that:
> 
>   1. Providing a migration period would be beneficial for our users, and
>   we should do that
>   2. Dropping deprecated Public APIs in minor releases, or demoting APIs
>   from Public to PublicEvolving / Experimental / Retired in any version bump,
>   are not good practices.
>   3. Ideally, with this FLIP, developers should be more careful and plan
>   API changes ahead. That means:
>      1. Be more careful with designing APIs and promoting them to Public,
>      so that they won't be changed / removed very soon, and also the
>      maintenance overhead for keeping them after deprecation should be
>      affordable. (I believe this also aligns with John's opinion.)
>      2. Plan the deprecation / removal of old APIs that we don't want to
>      carry to the next major version earlier, so that they'll have the
>      2-minor-release migration period before the next major version bump.
>   4. A practical situation is that, we are not ready for deprecating the
>   DataStream API, nor afford to carry it for another couple of years.
>      1. We cannot deprecate it now, because the new ProcessFunction API
>      (the planned replacement for DataStream API) is not yet ready. We haven't
>      planned the ProcessFunction API earlier because the original
> plan was to do
>      in-place changes directly on the DataStream API in release 2.0, until the
>      smooth migration period is proposed. Before this, the only guarantees we
>      provide for Public APIs is that they'll stay compatible until the next
>      major version, which led to the understanding the in-place changes on
>      Public APIs should be fine.
>      2. We also cannot afford carrying the DataStream API for another
>      couple of years. One of the reasons that we want to refactor / replace
>      DataStream API is that it exposes and depends on too many
> Flink's internal
>      runtime implementations, which has already limited / blocked plans for
>      Flink's internal improvements for many times. Also due to the wide and
>      transitive dependencies on the internals, maintaining two sets of lower
>      APIs itself is expensive and would be better to keep it short. This is
>      admittedly a historical design flaw which should be avoided in future.
> 
> What we haven't reached consensus is about how to deal with the current
> situation around the DataStream API. There are two options.
> 
> 
>   1. We say this FLIP is enforced starting release 2.0. For current 1.x
>   APIs, we provide a migration period with best effort, while allowing
>   exceptions for immediate removal in 2.0. That means we will still try with
>   best effort to get the ProcessFuncion API ready and deprecate the
>   DataStream API in 1.x, but will also be allowed to remove DataStream API in
>   2.0 if it's not deprecated 2 minor releases before the major version bump.
>   2. We strictly follow the process in this FLIP, and will quickly bump
>   the major version from 2.x to 3.0 once the migration period for DataStream
>   API is reached.
> 
> I'm personally in favor of option 1. As the migration-period rule is newly
> proposed, I think it's fair to make exceptions for cases where we already
> missed the best chance for planning the deprecation. Moreover, I do believe
> having a quick major version bump does confuse users. Yes, we can explain
> to users that bumping from 2.x to 3.0 does not cost anything other than the
> removal of an deprecated API. But having to explain this itself is an
> indicator that it might be confusing for users.
> 
> 
> Becket, on the other hand, prefers option 2. From my understanding, his
> major point is that a quick major version bump causes barely any actual
> lose on users, while in option 1 not providing the migration period or
> providing a shorter on is an actual lose on users. (@Becket, please correct
> me if I'm mistaken.)
> 
> 
> And we'd like to hear more feedback from the community.
> 
> 
> Best,
> 
> Xintong
> 
> 
> 
> On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcephei@apache.org <ma...@apache.org>> wrote:
> 
>> Hi Becket and Xintong,
>> 
>> I hope you don't mind if I chime in a little.
>> 
>> Once an API is marked Public, we're committing to support it until it's
>> deprecated, and once it's deprecated, to leave it in place for at least two
>> minor releases and then only remove it in the following major release.
>> 
>> As a user, I would be dismayed to discover that the project maintainers
>> demoted a Public API to PublicEvolving or Experimental in order to violate
>> the spirit of the deprecation period for Public APIs. Likewise, I'd view a
>> rapid sequence of minor releases for the purpose of dropping deprecated
>> APIs in the next major release as another violation of the spirit of our
>> guarantees. I'm reminded of this xkcd: https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
>> 
>> I'm glad we're gaming out these situations before we institute this FLIP,
>> and I hope we all converge on a clear understanding of what we're
>> guaranteeing. From where I'm sitting, the essence of this FLIP is
>> specifically to prevent us from doing whatever we want and instead to
>> commit to providing a suitable notice period to users who rely on Public
>> APIs.
>> 
>> It honestly shouldn't matter what the maintenance overhead is (or, rather,
>> evaluating the maintenance overhead should be a consideration before we
>> promote an API to Public to begin with). My experience in other projects is
>> that regretting a feature so much you want to drop it asap is extremely
>> rare, and that living with the pain for a little while until the
>> deprecation period expires will probably make us better engineers.
>> 
>> With that all said, one thing that is not a violation is to propose a new
>> major release in order to drop especially burdensome deprecated APIs that
>> have already enjoyed their full deprecation period. Those really bad "case
>> A" features will be very rare.
>> 
>> Looking over the FLIP again, I see a reference to the release
>> documentation, but not to the build tooling support for the Deprecated
>> annotation, so it might help to share this scenario: The way this will all
>> come together in practice is that, by giving good stability guarantees, we
>> will encourage our users to bump minor versions when they are released,
>> which means that they will see compiler warnings immediately if any of the
>> APIs they've used are deprecated. In fact, I'd recommend for them to use
>> the `-Werror` javac flag to be sure that they don't miss those warnings.
>> Given our release cadence, this means that they will have about six months
>> to migrate off of any deprecated APIs without being forced to avoid
>> updates. And they really need to be able to adopt further minor updates in
>> the series, since they may contain further improvements that actually
>> facilitate migrating off of the deprecated APIs.
>> 
>> The compiler warnings are a far more reliable mechanism to advertise
>> deprecation than release notes, since users' own builds will notify them
>> right away about only the deprecations that are relevant to them, but it
>> does rely on users feeling ok to bump up minor versions in a timely
>> fashion. If we consistently burn them by not observing the stability
>> guarantees, they'll try to avoid updates instead, and the whole thing falls
>> apart.
>> 
>> Thanks again,
>> -John
>> 
>> On 2023/06/19 10:43:05 Becket Qin wrote:
>>> Hi Xintong,
>>> 
>>> Let's compare the following cases:
>>> 
>>> A. If the maintenance overhead of the deprecated API is high, and we want
>>> to remove it after two minor releases. Then there are two options:
>>>    A1: Demote the API in the major version bump and remove the code
>> with a
>>> minor version bump.
>>>    A2: Do exactly the same as 1-A, except that when removing the code,
>>> bump up the major version. So this might be a short major version.
>>> 
>>> B. If the maintenance overhead of the deprecated API is not high,
>> therefore
>>> keeping it for a long time is affordable. There are also two options:
>>>    B1: Same as A-1, demote the API in the major version bump and remove
>>> the code with a minor version bump.
>>>    B2: Keep the API for all the minor versions in the major version, and
>>> only remove the code in the next major version.
>>> 
>>> For case B, do we agree that B2 is the way to go?
>>> 
>>> For case A:
>>> The following stays the same for A1 and A2:
>>>  - users will lose the API after two minor leases. So the migration
>> period
>>> is the same for A-1 and A-2.
>>>  - A1 and A2 will have exactly the same code after the removal of
>>> deprecated API, the only difference is versioning.
>>>  - To move forward, users need to move to the next minor version in A1,
>> or
>>> to the next major version in A2. Because the code are the same, the
>> actual
>>> effort to upgrade is the same for A1 and A2.
>>> 
>>> The differences between A-1 and A-2 are:
>>>  - A1 allows keeping the major version release cadence. A-2 will have a
>>> short major version release.
>>>  - A1 breaks the well understood API semantic, while A-2 does not.
>>> 
>>> From what I see, since there is no well establish standard regarding how
>>> long a major version should be released, A short major version release is
>>> potential and emotional. It is not ideal but does not have material
>>> downsides compared with A1. I did not hear anyone complaining about Kafka
>>> only has two 1.x release. However, A1 actually breaks the well understood
>>> API semantic, which has more material impact.
>>> 
>>> Also, I'd imagine 90% or more of the Public APIs should fall into case B.
>>> So, short major versions should be very occasional. I'd be very concerned
>>> if the reason we choose A1 is simply because we cannot afford
>> maintaining a
>>> bunch of deprecated APIs until the next major version. This indicates
>> that
>>> the actual problem we need to solve is to lower the maintenance overhead
>> of
>>> deprecated APIs, so that we are comfortable to keep them longer. As John
>>> and I mentioned earlier, there are ways to achieve this and we need to
>>> learn how to do it in Flink. Otherwise, our discussion about versioning
>>> here does not bring much value, because we will end up with a bunch of
>>> short-lived APIs which upset our users, no matter how we version the
>>> releases.
>>> 
>>> So, if there are concrete examples that you think will block us from
>>> keeping API stability with affordable cost, let's take a look together
>> and
>>> see if that can be improved.
>>> 
>>> Thanks,
>>> 
>>> Jiangjie (Becket) Qin
>>> 
>>> 
>>> From what I see,
>>> 
>>> 
>>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong820@gmail.com <ma...@gmail.com>>
>> wrote:
>>> 
>>>>> 
>>>>> The part I don't understand is if we are willing to have a migration
>>>>> period, and do a minor version bump to remove an API, what do we
>> lose to
>>>> do
>>>>> a major version bump instead, so we don't break the common versioning
>>>>> semantic?
>>>>> 
>>>> 
>>>> I think we are talking about the cases where a major version bump
>> happens
>>>> before a deprecated Public API reaches its migration period. So
>> removing
>>>> the API with another major version bump means we have two consecutive
>> major
>>>> versions in a very short time. And as previously mentioned, having
>> frequent
>>>> major version bumps would weaken the value of the commitment "Public
>> API
>>>> stay compatible within a major version". I think users also have
>>>> expectations about how long a major version should live, which should
>> be at
>>>> least a couple of years rather than 1-2 minor releases.
>>>> 
>>>> This is another option, but I think it is very likely more expensive
>> than
>>>>> simply bumping the major version. And I imagine there will be
>> questions
>>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
>>>>> 
>>>> 
>>>> Yes, it's more expensive than simply bumping the major version, but
>> IMHO
>>>> it's probably cheaper than carrying the API until the next major
>> version if
>>>> we don't bump the major version very soon. And see my reply above on
>> why
>>>> not bumping immediately.
>>>> 
>>>> Best,
>>>> 
>>>> Xintong
>>>> 
>>>> 
>>>> 
>>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket.qin@gmail.com <ma...@gmail.com>>
>> wrote:
>>>> 
>>>>> Hi Xintong,
>>>>> 
>>>>> Please see the replies below.
>>>>> 
>>>>> I see your point, that users would feel surprised if they find
>> things no
>>>>>> longer work when upgrading to another 2.x minor release. However,
>> I'd
>>>>> like
>>>>>> to point out that PublicEvolving APIs would have the similar
>> problem
>>>>>> anyway. So the question is, how do we catch users' attention and
>> make
>>>>> sure
>>>>>> they are aware that the Public APIs in 1.x may no longer be Public
>> in
>>>>> 2.0.
>>>>>> There're various ways to do that, e.g., release notes, warnings in
>>>> logs,
>>>>>> etc.
>>>>> 
>>>>> First of all, I am not a fan of removing PublicEvolving APIs in minor
>>>>> version changes. Personally speaking, I want them to be also removed
>> in
>>>>> major version changes.
>>>>> 
>>>>> Empirically, I rarely see projects with complex API semantic work
>> well.
>>>>> Many, if not most, users don't read docs / release notes / warnings
>> in
>>>>> logs. I don't think that is their fault. Typically an application
>>>> developer
>>>>> will have to deal with dozens of libraries maintained by various
>>>>> communities / groups. It is just too difficult for them to keep
>> track of
>>>>> all the specific semantics each project puts there. If you think
>> about
>>>> it,
>>>>> how many of Flink developers read all the release notes of Guava,
>> Apache
>>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of
>> them?
>>>>> One would probably try bumping the dependency version and see if
>> there is
>>>>> an exception and solve them case by case. And if it is a minor
>> version
>>>>> bump, one probably does not expect an exception at all. Another
>> example
>>>> is
>>>>> that in Flink we still have so many usage of deprecated methods all
>> over
>>>>> the place, and I strongly doubt everyone knows when the source code
>> of
>>>>> these methods are deprecated and when they should be removed.
>>>>> 
>>>>> So, the most important principle of API is simple and intuitive. The
>>>>> versioning semantic is a simple and universally accepted API
>> stability
>>>>> standard. If we ourselves as Flink developers are relying on this
>> for our
>>>>> own dependencies. I don't think we can expect more from our users.
>>>>> 
>>>>> Another possible alternative: whenever there's a deprecated Public
>> API
>>>> that
>>>>>> reaches a major version bump before the migration period, and we
>> also
>>>>> don't
>>>>>> want to carry it for all the next major release series, we may
>> consider
>>>>>> releasing more minor releases for the previous major version after
>> the
>>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
>> to
>>>> 2.0,
>>>>>> we can release one more 1.20 after 2.0. That should provide users
>>>> another
>>>>>> choice rather than upgrading to 2.0, while satisfying the
>>>> 2-minor-release
>>>>>> migration period.
>>>>> 
>>>>> This is another option, but I think it is very likely more expensive
>> than
>>>>> simply bumping the major version. And I imagine there will be
>> questions
>>>>> like "why this feature is in 1.20, but does not exist in 2.0"?
>>>>> 
>>>>> I think my major point is, we should not carry APIs deprecated in a
>>>>>> previous major version along all the next major version series. I'd
>>>> like
>>>>> to
>>>>>> try giving users more commitments, i.e. the migration period, as
>> long
>>>> as
>>>>> it
>>>>>> does not prevent us from making breaking changes. If it doesn't
>> work,
>>>> I'd
>>>>>> be in favor of not providing the migration period, but fallback to
>> only
>>>>>> guarantee the compatibility within the major version.
>>>>> 
>>>>> The part I don't understand is if we are willing to have a migration
>>>>> period, and do a minor version bump to remove an API, what do we
>> lose to
>>>> do
>>>>> a major version bump instead, so we don't break the common versioning
>>>>> semantic?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jiangjie (Becket) Qin
>>>>> 
>>>>> 
>>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <tonysong820@gmail.com <ma...@gmail.com>>
>>>>> wrote:
>>>>> 
>>>>>>> 
>>>>>>> As an end user who only uses Public APIs, if I don't change my
>> code
>>>> at
>>>>>>> all, my expectation is the following:
>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
>> with
>>>>> all
>>>>>>> the 2.x versions.
>>>>>>> Actually I think there are some dependency version resolution
>>>> policies
>>>>>> out
>>>>>>> there which picks the highest minor version when the dependencies
>>>> pull
>>>>> in
>>>>>>> multiple minor versions of the same jar, which may be broken if
>> we
>>>>> remove
>>>>>>> the API in minor releases.
>>>>>>> 
>>>>>> 
>>>>>> I see your point, that users would feel surprised if they find
>> things
>>>> no
>>>>>> longer work when upgrading to another 2.x minor release. However,
>> I'd
>>>>> like
>>>>>> to point out that PublicEvolving APIs would have the similar
>> problem
>>>>>> anyway. So the question is, how do we catch users' attention and
>> make
>>>>> sure
>>>>>> they are aware that the Public APIs in 1.x may no longer be Public
>> in
>>>>> 2.0.
>>>>>> There're various ways to do that, e.g., release notes, warnings in
>>>> logs,
>>>>>> etc.
>>>>>> 
>>>>>> Another possible alternative: whenever there's a deprecated Public
>> API
>>>>> that
>>>>>> reaches a major version bump before the migration period, and we
>> also
>>>>> don't
>>>>>> want to carry it for all the next major release series, we may
>> consider
>>>>>> releasing more minor releases for the previous major version after
>> the
>>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump
>> to
>>>> 2.0,
>>>>>> we can release one more 1.20 after 2.0. That should provide users
>>>> another
>>>>>> choice rather than upgrading to 2.0, while satisfying the
>>>> 2-minor-release
>>>>>> migration period.
>>>>>> 
>>>>>> I think my major point is, we should not carry APIs deprecated in a
>>>>>> previous major version along all the next major version series. I'd
>>>> like
>>>>> to
>>>>>> try giving users more commitments, i.e. the migration period, as
>> long
>>>> as
>>>>> it
>>>>>> does not prevent us from making breaking changes. If it doesn't
>> work,
>>>> I'd
>>>>>> be in favor of not providing the migration period, but fallback to
>> only
>>>>>> guarantee the compatibility within the major version.
>>>>>> 
>>>>>> Best,
>>>>>> 
>>>>>> Xintong
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcephei@apache.org <ma...@apache.org>
>>> 
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Becket,
>>>>>>> 
>>>>>>> Thanks for the reply! I’d like to continue the conversation about
>>>>>>> compatibility outside of this FLIP thread, but for now, I can
>> accept
>>>>> your
>>>>>>> decision. It’s certainly an improvement.
>>>>>>> 
>>>>>>> Thanks again,
>>>>>>> John
>>>>>>> 
>>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
>>>>>>>> Hi John,
>>>>>>>> 
>>>>>>>> Completely agree with all you said.
>>>>>>>> 
>>>>>>>> Can we consider only dropping deprecated APIs in major releases
>>>>> across
>>>>>>> the
>>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
>> are
>>>> by
>>>>>>>>> definition less stable, but it seems like this should be
>> reflected
>>>>> in
>>>>>>> the
>>>>>>>>> required deprecation period alone. I.e. that we must keep them
>>>>> around
>>>>>>> for
>>>>>>>>> at least zero or one minor release, not that we can drop them
>> in a
>>>>>>> minor or
>>>>>>>>> patch release.
>>>>>>>> 
>>>>>>>> Personally speaking, I would love to do this, for exactly the
>>>> reason
>>>>>> you
>>>>>>>> mentioned. However, I did not propose this due to the following
>>>>>> reasons:
>>>>>>>> 
>>>>>>>> 1. I am hesitating a little bit about changing the accepted
>> FLIPs
>>>> too
>>>>>>> soon.
>>>>>>>> 2. More importantly, to avoid slowing down our development. At
>> this
>>>>>>> point,
>>>>>>>> Flink still lacks some design / routines to support good API
>>>>>>> evolvability /
>>>>>>>> extensibility. Just like you said, it takes some time to be
>> good at
>>>>>> this.
>>>>>>>> In this case, my concern is that only removing Experimental /
>>>>>>>> PublicEvolving APIs in major version changes may result in too
>> much
>>>>>>>> overhead and dramatically slow down the development of Flink.
>> So, I
>>>>> was
>>>>>>>> thinking that we can start with the current status. Hopefully
>> after
>>>>> we
>>>>>>> are
>>>>>>>> more comfortable with the maintenance overhead of deprecated
>> APIs,
>>>> we
>>>>>> can
>>>>>>>> then have a stronger guarantee for Experimental /
>> PublicEvolving
>>>>> APIs.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jiangjie (Becket) Qin
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
>> vvcephei@apache.org <ma...@apache.org>>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Becket,
>>>>>>>>> 
>>>>>>>>> Thanks for this FLIP! Having a deprecation process is really
>>>>>> important.
>>>>>>> I
>>>>>>>>> understand some people’s concerns about the additional burden
>> for
>>>>>>> project
>>>>>>>>> maintainers, but my personal experience with Kafka has been
>> that
>>>>> it’s
>>>>>>> very
>>>>>>>>> liveable and that it’s well worth the benefit to users. In
>> fact,
>>>>> users
>>>>>>>>> being able to confidently upgrade is also a benefit to
>>>> maintainers,
>>>>> as
>>>>>>> we
>>>>>>>>> will get fewer questions from people stuck on very old
>> versions.
>>>>>>>>> 
>>>>>>>>> One question:
>>>>>>>>> Can we consider only dropping deprecated APIs in major
>> releases
>>>>> across
>>>>>>> the
>>>>>>>>> board? I understand that Experimental and PublicEvolving APIs
>> are
>>>> by
>>>>>>>>> definition less stable, but it seems like this should be
>> reflected
>>>>> in
>>>>>>> the
>>>>>>>>> required deprecation period alone. I.e. that we must keep them
>>>>> around
>>>>>>> for
>>>>>>>>> at least zero or one minor release, not that we can drop them
>> in a
>>>>>>> minor or
>>>>>>>>> patch release.
>>>>>>>>> 
>>>>>>>>> The advantage of forbidding the removal of any API in minor or
>>>> patch
>>>>>>>>> releases is that users will get a strong guarantee that they
>> can
>>>>> bump
>>>>>>> the
>>>>>>>>> minor or patch version and still be able to compile, or even
>> just
>>>>>>> re-link
>>>>>>>>> and know that they won’t face “MethodDef” exceptions at run
>> time.
>>>>> This
>>>>>>> is a
>>>>>>>>> binary guarantee: if we allow removing  even Experimental APIs
>>>>> outside
>>>>>>> of
>>>>>>>>> major releases, users can no longer confidently upgrade.
>>>>>>>>> 
>>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points:
>>>>>>>>> * I’d use the official Deprecated annotation instead of
>>>> introducing
>>>>>> our
>>>>>>>>> own flavor (Retired, etc), since Deprecated is well integrated
>>>> into
>>>>>>> build
>>>>>>>>> tools and IDEs.
>>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it
>> seems
>>>>>>>>> orthogonal, and something that should probably be taken
>>>> case-by-case
>>>>>>>>> anyway.
>>>>>>>>> * Aside from deprecation and removal, there have been some
>>>>> discussions
>>>>>>>>> about how to evolve APIs and behavior in compatible ways.
>> This is
>>>>>>> somewhat
>>>>>>>>> of an art, and if folks haven’t wrestled with it before, it’ll
>>>> take
>>>>>> some
>>>>>>>>> time to become good at it. I feel like this topic should also
>> be
>>>>>>> orthogonal
>>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a
>> simple
>>>>>> policy
>>>>>>> not
>>>>>>>>> to break existing user programs, and leave the “how” up to
>>>>>> implementers
>>>>>>> and
>>>>>>>>> reviewers.
>>>>>>>>> 
>>>>>>>>> Thanks again,
>>>>>>>>> John
>>>>>>>>> 
>>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a
>> great
>>>>> idea.
>>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
>>>> @PublicEvolving
>>>>>> --(2
>>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major
>>>>>>>>>> release)--> @PublicRetired. It will provide a lot of
>> flexibility
>>>>>>> without
>>>>>>>>>> breaking any rules we had. @Public APIs are allowed to
>> change
>>>>>> between
>>>>>>>>> major
>>>>>>>>>> releases. Changing annotations is acceptable and provides
>>>>> additional
>>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself
>> are
>>>> not
>>>>>>>>> changed.
>>>>>>>>>> 
>>>>>>>>>> I had similar thoughts when I was facing those issues. I
>> want to
>>>>>> move
>>>>>>> one
>>>>>>>>>> step further and suggest introducing one more annotation
>>>> @Retired.
>>>>>>>>>> 
>>>>>>>>>> Not like the @PublicRetired which is a compromise of
>> downgrading
>>>>>>> @Public
>>>>>>>>> to
>>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java
>>>> standard
>>>>>>>>>> @deprecated should be used in the early stage of the
>> deprecation
>>>>>>> process
>>>>>>>>>> and doesn't really meet our requirement. Since Java does not
>>>> allow
>>>>>> us
>>>>>>> to
>>>>>>>>>> extend annotation, I think it would be feasible to have the
>> new
>>>>>>> @Retired
>>>>>>>>> to
>>>>>>>>>> help us monitor and manage the deprecation process, house
>>>>> cleaning,
>>>>>>> etc.
>>>>>>>>>> 
>>>>>>>>>> Some ideas could be(open for discussion):
>>>>>>>>>> 
>>>>>>>>>> @Retired:
>>>>>>>>>> 
>>>>>>>>>> 1. There must be a replacement with functionality
>> compatibility
>>>>>> before
>>>>>>>>> APIs
>>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes
>> on
>>>> the
>>>>> ML
>>>>>>> are
>>>>>>>>>> mandatory (a FLIP is recommended).
>>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor
>> release
>>>>>>> sharply
>>>>>>>>>> (using ArchUnit to force it, needs further check whether it
>> is
>>>>>>> possible).
>>>>>>>>>> Devs who marked them as @Retired are responsible to remove
>> them.
>>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
>> are
>>>>>>>>>> recommended. @Experimental -> @Retired and @Internal ->
>> @Retired
>>>>>> could
>>>>>>>>> also
>>>>>>>>>> be used if it can increase user-friendliness or
>>>> dev-friendliness,
>>>>>> but
>>>>>>> not
>>>>>>>>>> mandatory.
>>>>>>>>>> 4. Some variables will be defined in @Retired to support the
>>>>>>> deprecation
>>>>>>>>>> process management. Further extension is possible, since the
>>>>>>> annotation
>>>>>>>>> is
>>>>>>>>>> built by us.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Best regards,
>>>>>>>>>> Jing
>>>>>>>>>> 
>>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
>>>> becket.qin@gmail.com <ma...@gmail.com>
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Xintong,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the explanation. Please see the replies inline
>>>> below.
>>>>>>>>>>> 
>>>>>>>>>>> I agree. And from my understanding, demoting a Public API
>> is
>>>>> also a
>>>>>>>>> kind of
>>>>>>>>>>>> such change, just like removing one, which can only
>> happen
>>>> with
>>>>>>> major
>>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
>>>> APIs
>>>>>>>>> anytime,
>>>>>>>>>>> but
>>>>>>>>>>>> only in the case major version bumps happen before
>> reaching
>>>> the
>>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
>> would
>>>> be a
>>>>>>> weaker
>>>>>>>>>>>> change compared to removing the API immediately upon
>> major
>>>>>> version
>>>>>>>>> bumps,
>>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
>>>>>> migration
>>>>>>>>>>> period.
>>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
>> sounds
>>>>>>> against
>>>>>>>>>>>> conventions, we may introduce a new annotation if
>> necessary,
>>>>>> e.g.,
>>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
>>>>>>>>>>> 
>>>>>>>>>>> As an end user who only uses Public APIs, if I don't
>> change my
>>>>> code
>>>>>>> at
>>>>>>>>> all,
>>>>>>>>>>> my expectation is the following:
>>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
>>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am
>> fine
>>>>>> with
>>>>>>> all
>>>>>>>>>>> the 2.x versions.
>>>>>>>>>>> Actually I think there are some dependency version
>> resolution
>>>>>>> policies
>>>>>>>>> out
>>>>>>>>>>> there which picks the highest minor version when the
>>>> dependencies
>>>>>>> pull
>>>>>>>>> in
>>>>>>>>>>> multiple minor versions of the same jar, which may be
>> broken if
>>>>> we
>>>>>>>>> remove
>>>>>>>>>>> the API in minor releases.
>>>>>>>>>>> 
>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that
>> we
>>>>> bump
>>>>>> up
>>>>>>>>> the
>>>>>>>>>>>> major version whenever a breaking change is needed.
>> However,
>>>>> this
>>>>>>> also
>>>>>>>>>>>> weakens the value of the commitment that public APIs will
>>>> stay
>>>>>>> stable
>>>>>>>>>>>> within the major release series, as the series can end
>>>> anytime.
>>>>>>> IMHO,
>>>>>>>>>>> short
>>>>>>>>>>>> major release series are not something "make the end
>> users
>>>>>> happy",
>>>>>>> but
>>>>>>>>>>>> backdoors that allow us as the developers to make
>> frequent
>>>>>> breaking
>>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
>> can
>>>>>> still
>>>>>>>>> have
>>>>>>>>>>>> longer major release series, while only allowing Public
>> APIs
>>>>>>>>> deprecated
>>>>>>>>>>> at
>>>>>>>>>>>> the end of the previous major version to be removed in
>> the
>>>> next
>>>>>>> major
>>>>>>>>>>>> version.
>>>>>>>>>>> 
>>>>>>>>>>> I totally agree that frequent major version bumps are not
>>>> ideal,
>>>>>> but
>>>>>>>>> here
>>>>>>>>>>> we are comparing it with a minor version bump which
>> removes a
>>>>>> Public
>>>>>>>>> API.
>>>>>>>>>>> So the context is that we have already decided to remove
>> this
>>>>>> Public
>>>>>>> API
>>>>>>>>>>> while keeping everything else backwards compatible. I
>> think a
>>>>> major
>>>>>>>>> version
>>>>>>>>>>> bump is a commonly understood signal here, compared with a
>>>> minor
>>>>>>> version
>>>>>>>>>>> change. From end users' perspective, for those who are not
>>>>>> impacted,
>>>>>>> in
>>>>>>>>>>> this case upgrading a major version is not necessarily more
>>>>>> involved
>>>>>>>>> than
>>>>>>>>>>> upgrading a minor version - both should be as smooth as a
>>>>>> dependency
>>>>>>>>>>> version change. For those who are impacted, they will lose
>> the
>>>>>> Public
>>>>>>>>> API
>>>>>>>>>>> anyways and a major version bump ensures there is no
>> surprise.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jiangjie (Becket) Qin
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
>>>>>> tonysong820@gmail.com <ma...@gmail.com>
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Public API is a well defined common concept, and one of
>> its
>>>>>>>>>>>>> convention is that it only changes with a major version
>>>>> change.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I agree. And from my understanding, demoting a Public
>> API is
>>>>>> also a
>>>>>>>>> kind
>>>>>>>>>>>> of such change, just like removing one, which can only
>> happen
>>>>>> with
>>>>>>>>> major
>>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public
>>>> APIs
>>>>>>>>> anytime,
>>>>>>>>>>> but
>>>>>>>>>>>> only in the case major version bumps happen before
>> reaching
>>>> the
>>>>>>>>>>>> 2-minor-release migration period. Actually, demoting
>> would
>>>> be a
>>>>>>> weaker
>>>>>>>>>>>> change compared to removing the API immediately upon
>> major
>>>>>> version
>>>>>>>>> bumps,
>>>>>>>>>>>> in order to keep the commitment about the 2-minor-release
>>>>>> migration
>>>>>>>>>>> period.
>>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving`
>> sounds
>>>>>>> against
>>>>>>>>>>>> conventions, we may introduce a new annotation if
>> necessary,
>>>>>> e.g.,
>>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
>>>>>>>>>>>> 
>>>>>>>>>>>> But it should be
>>>>>>>>>>>>> completely OK to bump up the major version if we really
>> want
>>>>> to
>>>>>>> get
>>>>>>>>> rid
>>>>>>>>>>> of
>>>>>>>>>>>>> a public API, right?
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal"
>> that we
>>>>>> bump
>>>>>>> up
>>>>>>>>> the
>>>>>>>>>>>> major version whenever a breaking change is needed.
>> However,
>>>>> this
>>>>>>> also
>>>>>>>>>>>> weakens the value of the commitment that public APIs will
>>>> stay
>>>>>>> stable
>>>>>>>>>>>> within the major release series, as the series can end
>>>> anytime.
>>>>>>> IMHO,
>>>>>>>>>>> short
>>>>>>>>>>>> major release series are not something "make the end
>> users
>>>>>> happy",
>>>>>>> but
>>>>>>>>>>>> backdoors that allow us as the developers to make
>> frequent
>>>>>> breaking
>>>>>>>>>>>> changes. On the contrary, with the demoting approach, we
>> can
>>>>>> still
>>>>>>>>> have
>>>>>>>>>>>> longer major release series, while only allowing Public
>> APIs
>>>>>>>>> deprecated
>>>>>>>>>>> at
>>>>>>>>>>>> the end of the previous major version to be removed in
>> the
>>>> next
>>>>>>> major
>>>>>>>>>>>> version.
>>>>>>>>>>>> 
>>>>>>>>>>>> Given our track record I would prefer a regular cycle
>> (1-2
>>>>> years)
>>>>>>> to
>>>>>>>>>>>>> force us to think about this whole topic, and not put it
>>>> again
>>>>>> to
>>>>>>> the
>>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
>>>> when
>>>>>>>>> breaking
>>>>>>>>>>>>> changes can be made.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> +1. I personally think 2-3 years would be a good time
>> for new
>>>>>> major
>>>>>>>>>>>> versions, or longer if there's no breaking changes
>> needed.
>>>> That
>>>>>>> makes
>>>>>>>>> 1-2
>>>>>>>>>>>> year a perfect time to revisit the topic, while leaving
>> us
>>>> more
>>>>>>> time
>>>>>>>>> to
>>>>>>>>>>>> prepare the major release if needed.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> 
>>>>>>>>>>>> Xintong
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
>>>>>>> chesnay@apache.org <ma...@apache.org>
>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
>>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor
>>>>> versions
>>>>>>> for
>>>>>>>>>>>>> previous
>>>>>>>>>>>>>> major versions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how
>> viable
>>>> that
>>>>>> is
>>>>>>> in
>>>>>>>>>>>>> practice.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping
>> support
>>>> for
>>>>>> Java
>>>>>>>>> 8/11,
>>>>>>>>>>>>> which may very well be a problem for our current users.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
>>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion
>> about
>>>>>> Public
>>>>>>>>> API
>>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake
>> in
>>>> the
>>>>>>> NOTES
>>>>>>>>> part
>>>>>>>>>>>>> for
>>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed
>> it.
>>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we
>> are on
>>>>> the
>>>>>>> same
>>>>>>>>>>>>> page on that note.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
>>>>>>>>>>>>>> But it should be
>>>>>>>>>>>>>> completely OK to bump up the major version if we
>> really
>>>> want
>>>>>> to
>>>>>>> get
>>>>>>>>>>> rid
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> a public API, right?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Technically yes, but look at how long it took to get us
>> to
>>>>> 2.0.
>>>>>> ;)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There's a separate discussion to be had on the cadence
>> of
>>>>> major
>>>>>>>>> releases
>>>>>>>>>>>>> going forward, and there seem to be different opinions
>> on
>>>>> that.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between
>>>> major
>>>>>>> ones,
>>>>>>>>>>>>> that for us means that users have to potentially deal
>> with
>>>>>>> breaking
>>>>>>>>>>>>> changes every 6 months, which seems like a lot.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Given our track record I would prefer a regular cycle
>> (1-2
>>>>>> years)
>>>>>>> to
>>>>>>>>>>>>> force us to think about this whole topic, and not put it
>>>> again
>>>>>> to
>>>>>>> the
>>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on
>>>> when
>>>>>>>>> breaking
>>>>>>>>>>>>> changes can be made.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> But again, maybe this should be in a separate thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
>>>>>>>>>>>>>> Do you have an example of behavioral change in mind?
>> Not
>>>>> sure
>>>>>> I
>>>>>>>>> fully
>>>>>>>>>>>>>> understand the concern for behavioral change here.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This could be a lot of things. It can be performance in
>>>>> certain
>>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly)
>> relied
>>>>> upon
>>>>>>>>>>>>> (https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I), a semantic change to some
>> API.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For a concrete example, consider the job submission. A
>> few
>>>>>>> releases
>>>>>>>>> back
>>>>>>>>>>>>> we made changes such that the initialization of the job
>>>> master
>>>>>>>>> happens
>>>>>>>>>>>>> asynchronously.
>>>>>>>>>>>>> This meant the job submission call returns sooner, and
>> the
>>>> job
>>>>>>> state
>>>>>>>>>>>>> enum was extended to cover this state.
>>>>>>>>>>>>> API-wise we consider this a compatible change, but the
>>>>> observed
>>>>>>>>> behavior
>>>>>>>>>>>>> may be different.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Metrics are another example; I believe over time we
>> changed
>>>>> what
>>>>>>> some
>>>>>>>>>>>>> metrics returned a few times.


Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
Becket and I had an offline voice call earlier today. We have reached
consensus on some of the arguments, but still hold different opinions on
some others. I'd like to post the outcome here for transparency.

We both agree that:

   1. Providing a migration period would be beneficial for our users, and
   we should do that
   2. Dropping deprecated Public APIs in minor releases, or demoting APIs
   from Public to PublicEvolving / Experimental / Retired in any version bump,
   are not good practices.
   3. Ideally, with this FLIP, developers should be more careful and plan
   API changes ahead. That means:
      1. Be more careful with designing APIs and promoting them to Public,
      so that they won't be changed / removed very soon, and also the
      maintenance overhead for keeping them after deprecation should be
      affordable. (I believe this also aligns with John's opinion.)
      2. Plan the deprecation / removal of old APIs that we don't want to
      carry to the next major version earlier, so that they'll have the
      2-minor-release migration period before the next major version bump.
   4. A practical situation is that, we are not ready for deprecating the
   DataStream API, nor afford to carry it for another couple of years.
      1. We cannot deprecate it now, because the new ProcessFunction API
      (the planned replacement for DataStream API) is not yet ready. We haven't
      planned the ProcessFunction API earlier because the original
plan was to do
      in-place changes directly on the DataStream API in release 2.0, until the
      smooth migration period is proposed. Before this, the only guarantees we
      provide for Public APIs is that they'll stay compatible until the next
      major version, which led to the understanding the in-place changes on
      Public APIs should be fine.
      2. We also cannot afford carrying the DataStream API for another
      couple of years. One of the reasons that we want to refactor / replace
      DataStream API is that it exposes and depends on too many
Flink's internal
      runtime implementations, which has already limited / blocked plans for
      Flink's internal improvements for many times. Also due to the wide and
      transitive dependencies on the internals, maintaining two sets of lower
      APIs itself is expensive and would be better to keep it short. This is
      admittedly a historical design flaw which should be avoided in future.

What we haven't reached consensus is about how to deal with the current
situation around the DataStream API. There are two options.


   1. We say this FLIP is enforced starting release 2.0. For current 1.x
   APIs, we provide a migration period with best effort, while allowing
   exceptions for immediate removal in 2.0. That means we will still try with
   best effort to get the ProcessFuncion API ready and deprecate the
   DataStream API in 1.x, but will also be allowed to remove DataStream API in
   2.0 if it's not deprecated 2 minor releases before the major version bump.
   2. We strictly follow the process in this FLIP, and will quickly bump
   the major version from 2.x to 3.0 once the migration period for DataStream
   API is reached.

I'm personally in favor of option 1. As the migration-period rule is newly
proposed, I think it's fair to make exceptions for cases where we already
missed the best chance for planning the deprecation. Moreover, I do believe
having a quick major version bump does confuse users. Yes, we can explain
to users that bumping from 2.x to 3.0 does not cost anything other than the
removal of an deprecated API. But having to explain this itself is an
indicator that it might be confusing for users.


Becket, on the other hand, prefers option 2. From my understanding, his
major point is that a quick major version bump causes barely any actual
lose on users, while in option 1 not providing the migration period or
providing a shorter on is an actual lose on users. (@Becket, please correct
me if I'm mistaken.)


And we'd like to hear more feedback from the community.


Best,

Xintong



On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vv...@apache.org> wrote:

> Hi Becket and Xintong,
>
> I hope you don't mind if I chime in a little.
>
> Once an API is marked Public, we're committing to support it until it's
> deprecated, and once it's deprecated, to leave it in place for at least two
> minor releases and then only remove it in the following major release.
>
> As a user, I would be dismayed to discover that the project maintainers
> demoted a Public API to PublicEvolving or Experimental in order to violate
> the spirit of the deprecation period for Public APIs. Likewise, I'd view a
> rapid sequence of minor releases for the purpose of dropping deprecated
> APIs in the next major release as another violation of the spirit of our
> guarantees. I'm reminded of this xkcd: https://xkcd.com/1494/
>
> I'm glad we're gaming out these situations before we institute this FLIP,
> and I hope we all converge on a clear understanding of what we're
> guaranteeing. From where I'm sitting, the essence of this FLIP is
> specifically to prevent us from doing whatever we want and instead to
> commit to providing a suitable notice period to users who rely on Public
> APIs.
>
> It honestly shouldn't matter what the maintenance overhead is (or, rather,
> evaluating the maintenance overhead should be a consideration before we
> promote an API to Public to begin with). My experience in other projects is
> that regretting a feature so much you want to drop it asap is extremely
> rare, and that living with the pain for a little while until the
> deprecation period expires will probably make us better engineers.
>
> With that all said, one thing that is not a violation is to propose a new
> major release in order to drop especially burdensome deprecated APIs that
> have already enjoyed their full deprecation period. Those really bad "case
> A" features will be very rare.
>
> Looking over the FLIP again, I see a reference to the release
> documentation, but not to the build tooling support for the Deprecated
> annotation, so it might help to share this scenario: The way this will all
> come together in practice is that, by giving good stability guarantees, we
> will encourage our users to bump minor versions when they are released,
> which means that they will see compiler warnings immediately if any of the
> APIs they've used are deprecated. In fact, I'd recommend for them to use
> the `-Werror` javac flag to be sure that they don't miss those warnings.
> Given our release cadence, this means that they will have about six months
> to migrate off of any deprecated APIs without being forced to avoid
> updates. And they really need to be able to adopt further minor updates in
> the series, since they may contain further improvements that actually
> facilitate migrating off of the deprecated APIs.
>
> The compiler warnings are a far more reliable mechanism to advertise
> deprecation than release notes, since users' own builds will notify them
> right away about only the deprecations that are relevant to them, but it
> does rely on users feeling ok to bump up minor versions in a timely
> fashion. If we consistently burn them by not observing the stability
> guarantees, they'll try to avoid updates instead, and the whole thing falls
> apart.
>
> Thanks again,
> -John
>
> On 2023/06/19 10:43:05 Becket Qin wrote:
> > Hi Xintong,
> >
> > Let's compare the following cases:
> >
> > A. If the maintenance overhead of the deprecated API is high, and we want
> > to remove it after two minor releases. Then there are two options:
> >     A1: Demote the API in the major version bump and remove the code
> with a
> > minor version bump.
> >     A2: Do exactly the same as 1-A, except that when removing the code,
> > bump up the major version. So this might be a short major version.
> >
> > B. If the maintenance overhead of the deprecated API is not high,
> therefore
> > keeping it for a long time is affordable. There are also two options:
> >     B1: Same as A-1, demote the API in the major version bump and remove
> > the code with a minor version bump.
> >     B2: Keep the API for all the minor versions in the major version, and
> > only remove the code in the next major version.
> >
> > For case B, do we agree that B2 is the way to go?
> >
> > For case A:
> > The following stays the same for A1 and A2:
> >   - users will lose the API after two minor leases. So the migration
> period
> > is the same for A-1 and A-2.
> >   - A1 and A2 will have exactly the same code after the removal of
> > deprecated API, the only difference is versioning.
> >   - To move forward, users need to move to the next minor version in A1,
> or
> > to the next major version in A2. Because the code are the same, the
> actual
> > effort to upgrade is the same for A1 and A2.
> >
> > The differences between A-1 and A-2 are:
> >   - A1 allows keeping the major version release cadence. A-2 will have a
> > short major version release.
> >   - A1 breaks the well understood API semantic, while A-2 does not.
> >
> > From what I see, since there is no well establish standard regarding how
> > long a major version should be released, A short major version release is
> > potential and emotional. It is not ideal but does not have material
> > downsides compared with A1. I did not hear anyone complaining about Kafka
> > only has two 1.x release. However, A1 actually breaks the well understood
> > API semantic, which has more material impact.
> >
> > Also, I'd imagine 90% or more of the Public APIs should fall into case B.
> > So, short major versions should be very occasional. I'd be very concerned
> > if the reason we choose A1 is simply because we cannot afford
> maintaining a
> > bunch of deprecated APIs until the next major version. This indicates
> that
> > the actual problem we need to solve is to lower the maintenance overhead
> of
> > deprecated APIs, so that we are comfortable to keep them longer. As John
> > and I mentioned earlier, there are ways to achieve this and we need to
> > learn how to do it in Flink. Otherwise, our discussion about versioning
> > here does not bring much value, because we will end up with a bunch of
> > short-lived APIs which upset our users, no matter how we version the
> > releases.
> >
> > So, if there are concrete examples that you think will block us from
> > keeping API stability with affordable cost, let's take a look together
> and
> > see if that can be improved.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > From what I see,
> >
> >
> > On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <to...@gmail.com>
> wrote:
> >
> > > >
> > > > The part I don't understand is if we are willing to have a migration
> > > > period, and do a minor version bump to remove an API, what do we
> lose to
> > > do
> > > > a major version bump instead, so we don't break the common versioning
> > > > semantic?
> > > >
> > >
> > > I think we are talking about the cases where a major version bump
> happens
> > > before a deprecated Public API reaches its migration period. So
> removing
> > > the API with another major version bump means we have two consecutive
> major
> > > versions in a very short time. And as previously mentioned, having
> frequent
> > > major version bumps would weaken the value of the commitment "Public
> API
> > > stay compatible within a major version". I think users also have
> > > expectations about how long a major version should live, which should
> be at
> > > least a couple of years rather than 1-2 minor releases.
> > >
> > > This is another option, but I think it is very likely more expensive
> than
> > > > simply bumping the major version. And I imagine there will be
> questions
> > > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > > >
> > >
> > > Yes, it's more expensive than simply bumping the major version, but
> IMHO
> > > it's probably cheaper than carrying the API until the next major
> version if
> > > we don't bump the major version very soon. And see my reply above on
> why
> > > not bumping immediately.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > > Hi Xintong,
> > > >
> > > > Please see the replies below.
> > > >
> > > > I see your point, that users would feel surprised if they find
> things no
> > > > > longer work when upgrading to another 2.x minor release. However,
> I'd
> > > > like
> > > > > to point out that PublicEvolving APIs would have the similar
> problem
> > > > > anyway. So the question is, how do we catch users' attention and
> make
> > > > sure
> > > > > they are aware that the Public APIs in 1.x may no longer be Public
> in
> > > > 2.0.
> > > > > There're various ways to do that, e.g., release notes, warnings in
> > > logs,
> > > > > etc.
> > > >
> > > > First of all, I am not a fan of removing PublicEvolving APIs in minor
> > > > version changes. Personally speaking, I want them to be also removed
> in
> > > > major version changes.
> > > >
> > > > Empirically, I rarely see projects with complex API semantic work
> well.
> > > > Many, if not most, users don't read docs / release notes / warnings
> in
> > > > logs. I don't think that is their fault. Typically an application
> > > developer
> > > > will have to deal with dozens of libraries maintained by various
> > > > communities / groups. It is just too difficult for them to keep
> track of
> > > > all the specific semantics each project puts there. If you think
> about
> > > it,
> > > > how many of Flink developers read all the release notes of Guava,
> Apache
> > > > Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of
> them?
> > > > One would probably try bumping the dependency version and see if
> there is
> > > > an exception and solve them case by case. And if it is a minor
> version
> > > > bump, one probably does not expect an exception at all. Another
> example
> > > is
> > > > that in Flink we still have so many usage of deprecated methods all
> over
> > > > the place, and I strongly doubt everyone knows when the source code
> of
> > > > these methods are deprecated and when they should be removed.
> > > >
> > > > So, the most important principle of API is simple and intuitive. The
> > > > versioning semantic is a simple and universally accepted API
> stability
> > > > standard. If we ourselves as Flink developers are relying on this
> for our
> > > > own dependencies. I don't think we can expect more from our users.
> > > >
> > > > Another possible alternative: whenever there's a deprecated Public
> API
> > > that
> > > > > reaches a major version bump before the migration period, and we
> also
> > > > don't
> > > > > want to carry it for all the next major release series, we may
> consider
> > > > > releasing more minor releases for the previous major version after
> the
> > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump
> to
> > > 2.0,
> > > > > we can release one more 1.20 after 2.0. That should provide users
> > > another
> > > > > choice rather than upgrading to 2.0, while satisfying the
> > > 2-minor-release
> > > > > migration period.
> > > >
> > > > This is another option, but I think it is very likely more expensive
> than
> > > > simply bumping the major version. And I imagine there will be
> questions
> > > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > > >
> > > > I think my major point is, we should not carry APIs deprecated in a
> > > > > previous major version along all the next major version series. I'd
> > > like
> > > > to
> > > > > try giving users more commitments, i.e. the migration period, as
> long
> > > as
> > > > it
> > > > > does not prevent us from making breaking changes. If it doesn't
> work,
> > > I'd
> > > > > be in favor of not providing the migration period, but fallback to
> only
> > > > > guarantee the compatibility within the major version.
> > > >
> > > > The part I don't understand is if we are willing to have a migration
> > > > period, and do a minor version bump to remove an API, what do we
> lose to
> > > do
> > > > a major version bump instead, so we don't break the common versioning
> > > > semantic?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > > On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com>
> > > > wrote:
> > > >
> > > > > >
> > > > > > As an end user who only uses Public APIs, if I don't change my
> code
> > > at
> > > > > > all, my expectation is the following:
> > > > > > 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> with
> > > > all
> > > > > > the 2.x versions.
> > > > > > Actually I think there are some dependency version resolution
> > > policies
> > > > > out
> > > > > > there which picks the highest minor version when the dependencies
> > > pull
> > > > in
> > > > > > multiple minor versions of the same jar, which may be broken if
> we
> > > > remove
> > > > > > the API in minor releases.
> > > > > >
> > > > >
> > > > > I see your point, that users would feel surprised if they find
> things
> > > no
> > > > > longer work when upgrading to another 2.x minor release. However,
> I'd
> > > > like
> > > > > to point out that PublicEvolving APIs would have the similar
> problem
> > > > > anyway. So the question is, how do we catch users' attention and
> make
> > > > sure
> > > > > they are aware that the Public APIs in 1.x may no longer be Public
> in
> > > > 2.0.
> > > > > There're various ways to do that, e.g., release notes, warnings in
> > > logs,
> > > > > etc.
> > > > >
> > > > > Another possible alternative: whenever there's a deprecated Public
> API
> > > > that
> > > > > reaches a major version bump before the migration period, and we
> also
> > > > don't
> > > > > want to carry it for all the next major release series, we may
> consider
> > > > > releasing more minor releases for the previous major version after
> the
> > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump
> to
> > > 2.0,
> > > > > we can release one more 1.20 after 2.0. That should provide users
> > > another
> > > > > choice rather than upgrading to 2.0, while satisfying the
> > > 2-minor-release
> > > > > migration period.
> > > > >
> > > > > I think my major point is, we should not carry APIs deprecated in a
> > > > > previous major version along all the next major version series. I'd
> > > like
> > > > to
> > > > > try giving users more commitments, i.e. the migration period, as
> long
> > > as
> > > > it
> > > > > does not prevent us from making breaking changes. If it doesn't
> work,
> > > I'd
> > > > > be in favor of not providing the migration period, but fallback to
> only
> > > > > guarantee the compatibility within the major version.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcephei@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > Hi Becket,
> > > > > >
> > > > > > Thanks for the reply! I’d like to continue the conversation about
> > > > > > compatibility outside of this FLIP thread, but for now, I can
> accept
> > > > your
> > > > > > decision. It’s certainly an improvement.
> > > > > >
> > > > > > Thanks again,
> > > > > > John
> > > > > >
> > > > > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Completely agree with all you said.
> > > > > > >
> > > > > > > Can we consider only dropping deprecated APIs in major releases
> > > > across
> > > > > > the
> > > > > > >> board? I understand that Experimental and PublicEvolving APIs
> are
> > > by
> > > > > > >> definition less stable, but it seems like this should be
> reflected
> > > > in
> > > > > > the
> > > > > > >> required deprecation period alone. I.e. that we must keep them
> > > > around
> > > > > > for
> > > > > > >> at least zero or one minor release, not that we can drop them
> in a
> > > > > > minor or
> > > > > > >> patch release.
> > > > > > >
> > > > > > > Personally speaking, I would love to do this, for exactly the
> > > reason
> > > > > you
> > > > > > > mentioned. However, I did not propose this due to the following
> > > > > reasons:
> > > > > > >
> > > > > > > 1. I am hesitating a little bit about changing the accepted
> FLIPs
> > > too
> > > > > > soon.
> > > > > > > 2. More importantly, to avoid slowing down our development. At
> this
> > > > > > point,
> > > > > > > Flink still lacks some design / routines to support good API
> > > > > > evolvability /
> > > > > > > extensibility. Just like you said, it takes some time to be
> good at
> > > > > this.
> > > > > > > In this case, my concern is that only removing Experimental /
> > > > > > > PublicEvolving APIs in major version changes may result in too
> much
> > > > > > > overhead and dramatically slow down the development of Flink.
> So, I
> > > > was
> > > > > > > thinking that we can start with the current status. Hopefully
> after
> > > > we
> > > > > > are
> > > > > > > more comfortable with the maintenance overhead of deprecated
> APIs,
> > > we
> > > > > can
> > > > > > > then have a stronger guarantee for Experimental /
> PublicEvolving
> > > > APIs.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> vvcephei@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Becket,
> > > > > > >>
> > > > > > >> Thanks for this FLIP! Having a deprecation process is really
> > > > > important.
> > > > > > I
> > > > > > >> understand some people’s concerns about the additional burden
> for
> > > > > > project
> > > > > > >> maintainers, but my personal experience with Kafka has been
> that
> > > > it’s
> > > > > > very
> > > > > > >> liveable and that it’s well worth the benefit to users. In
> fact,
> > > > users
> > > > > > >> being able to confidently upgrade is also a benefit to
> > > maintainers,
> > > > as
> > > > > > we
> > > > > > >> will get fewer questions from people stuck on very old
> versions.
> > > > > > >>
> > > > > > >> One question:
> > > > > > >> Can we consider only dropping deprecated APIs in major
> releases
> > > > across
> > > > > > the
> > > > > > >> board? I understand that Experimental and PublicEvolving APIs
> are
> > > by
> > > > > > >> definition less stable, but it seems like this should be
> reflected
> > > > in
> > > > > > the
> > > > > > >> required deprecation period alone. I.e. that we must keep them
> > > > around
> > > > > > for
> > > > > > >> at least zero or one minor release, not that we can drop them
> in a
> > > > > > minor or
> > > > > > >> patch release.
> > > > > > >>
> > > > > > >> The advantage of forbidding the removal of any API in minor or
> > > patch
> > > > > > >> releases is that users will get a strong guarantee that they
> can
> > > > bump
> > > > > > the
> > > > > > >> minor or patch version and still be able to compile, or even
> just
> > > > > > re-link
> > > > > > >> and know that they won’t face “MethodDef” exceptions at run
> time.
> > > > This
> > > > > > is a
> > > > > > >> binary guarantee: if we allow removing  even Experimental APIs
> > > > outside
> > > > > > of
> > > > > > >> major releases, users can no longer confidently upgrade.
> > > > > > >>
> > > > > > >> Aside from that, I’d share my 2 cents on a couple of points:
> > > > > > >> * I’d use the official Deprecated annotation instead of
> > > introducing
> > > > > our
> > > > > > >> own flavor (Retired, etc), since Deprecated is well integrated
> > > into
> > > > > > build
> > > > > > >> tools and IDEs.
> > > > > > >> * I wouldn’t worry about a demotion process in this FLIP; it
> seems
> > > > > > >> orthogonal, and something that should probably be taken
> > > case-by-case
> > > > > > >> anyway.
> > > > > > >> * Aside from deprecation and removal, there have been some
> > > > discussions
> > > > > > >> about how to evolve APIs and behavior in compatible ways.
> This is
> > > > > > somewhat
> > > > > > >> of an art, and if folks haven’t wrestled with it before, it’ll
> > > take
> > > > > some
> > > > > > >> time to become good at it. I feel like this topic should also
> be
> > > > > > orthogonal
> > > > > > >> to this FLIP, but FWIW, my suggestion would be to adopt a
> simple
> > > > > policy
> > > > > > not
> > > > > > >> to break existing user programs, and leave the “how” up to
> > > > > implementers
> > > > > > and
> > > > > > >> reviewers.
> > > > > > >>
> > > > > > >> Thanks again,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > >> > Hi All,
> > > > > > >> >
> > > > > > >> > The @Public -> @PublicEvolving proposed by Xintong is a
> great
> > > > idea.
> > > > > > >> > Especially, after he suggest @PublicRetired, i.e.
> > > @PublicEvolving
> > > > > --(2
> > > > > > >> > minor release)--> @Public --> @deprecated --(1 major
> > > > > > >> > release)--> @PublicRetired. It will provide a lot of
> flexibility
> > > > > > without
> > > > > > >> > breaking any rules we had. @Public APIs are allowed to
> change
> > > > > between
> > > > > > >> major
> > > > > > >> > releases. Changing annotations is acceptable and provides
> > > > additional
> > > > > > >> > tolerance i.e. user-friendliness, since the APIs themself
> are
> > > not
> > > > > > >> changed.
> > > > > > >> >
> > > > > > >> > I had similar thoughts when I was facing those issues. I
> want to
> > > > > move
> > > > > > one
> > > > > > >> > step further and suggest introducing one more annotation
> > > @Retired.
> > > > > > >> >
> > > > > > >> > Not like the @PublicRetired which is a compromise of
> downgrading
> > > > > > @Public
> > > > > > >> to
> > > > > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java
> > > standard
> > > > > > >> > @deprecated should be used in the early stage of the
> deprecation
> > > > > > process
> > > > > > >> > and doesn't really meet our requirement. Since Java does not
> > > allow
> > > > > us
> > > > > > to
> > > > > > >> > extend annotation, I think it would be feasible to have the
> new
> > > > > > @Retired
> > > > > > >> to
> > > > > > >> > help us monitor and manage the deprecation process, house
> > > > cleaning,
> > > > > > etc.
> > > > > > >> >
> > > > > > >> > Some ideas could be(open for discussion):
> > > > > > >> >
> > > > > > >> > @Retired:
> > > > > > >> >
> > > > > > >> > 1. There must be a replacement with functionality
> compatibility
> > > > > before
> > > > > > >> APIs
> > > > > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes
> on
> > > the
> > > > ML
> > > > > > are
> > > > > > >> > mandatory (a FLIP is recommended).
> > > > > > >> > 2. APIs marked as @Retired will be removed after 1 minor
> release
> > > > > > sharply
> > > > > > >> > (using ArchUnit to force it, needs further check whether it
> is
> > > > > > possible).
> > > > > > >> > Devs who marked them as @Retired are responsible to remove
> them.
> > > > > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired
> are
> > > > > > >> > recommended. @Experimental -> @Retired and @Internal ->
> @Retired
> > > > > could
> > > > > > >> also
> > > > > > >> > be used if it can increase user-friendliness or
> > > dev-friendliness,
> > > > > but
> > > > > > not
> > > > > > >> > mandatory.
> > > > > > >> > 4. Some variables will be defined in @Retired to support the
> > > > > > deprecation
> > > > > > >> > process management. Further extension is possible, since the
> > > > > > annotation
> > > > > > >> is
> > > > > > >> > built by us.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Best regards,
> > > > > > >> > Jing
> > > > > > >> >
> > > > > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > becket.qin@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> >> Hi Xintong,
> > > > > > >> >>
> > > > > > >> >> Thanks for the explanation. Please see the replies inline
> > > below.
> > > > > > >> >>
> > > > > > >> >> I agree. And from my understanding, demoting a Public API
> is
> > > > also a
> > > > > > >> kind of
> > > > > > >> >> > such change, just like removing one, which can only
> happen
> > > with
> > > > > > major
> > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > > APIs
> > > > > > >> anytime,
> > > > > > >> >> but
> > > > > > >> >> > only in the case major version bumps happen before
> reaching
> > > the
> > > > > > >> >> > 2-minor-release migration period. Actually, demoting
> would
> > > be a
> > > > > > weaker
> > > > > > >> >> > change compared to removing the API immediately upon
> major
> > > > > version
> > > > > > >> bumps,
> > > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > > migration
> > > > > > >> >> period.
> > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving`
> sounds
> > > > > > against
> > > > > > >> >> > conventions, we may introduce a new annotation if
> necessary,
> > > > > e.g.,
> > > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > > >> >>
> > > > > > >> >> As an end user who only uses Public APIs, if I don't
> change my
> > > > code
> > > > > > at
> > > > > > >> all,
> > > > > > >> >> my expectation is the following:
> > > > > > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am
> fine
> > > > > with
> > > > > > all
> > > > > > >> >> the 2.x versions.
> > > > > > >> >> Actually I think there are some dependency version
> resolution
> > > > > > policies
> > > > > > >> out
> > > > > > >> >> there which picks the highest minor version when the
> > > dependencies
> > > > > > pull
> > > > > > >> in
> > > > > > >> >> multiple minor versions of the same jar, which may be
> broken if
> > > > we
> > > > > > >> remove
> > > > > > >> >> the API in minor releases.
> > > > > > >> >>
> > > > > > >> >> I'm not sure about this. Yes, it's completely "legal" that
> we
> > > > bump
> > > > > up
> > > > > > >> the
> > > > > > >> >> > major version whenever a breaking change is needed.
> However,
> > > > this
> > > > > > also
> > > > > > >> >> > weakens the value of the commitment that public APIs will
> > > stay
> > > > > > stable
> > > > > > >> >> > within the major release series, as the series can end
> > > anytime.
> > > > > > IMHO,
> > > > > > >> >> short
> > > > > > >> >> > major release series are not something "make the end
> users
> > > > > happy",
> > > > > > but
> > > > > > >> >> > backdoors that allow us as the developers to make
> frequent
> > > > > breaking
> > > > > > >> >> > changes. On the contrary, with the demoting approach, we
> can
> > > > > still
> > > > > > >> have
> > > > > > >> >> > longer major release series, while only allowing Public
> APIs
> > > > > > >> deprecated
> > > > > > >> >> at
> > > > > > >> >> > the end of the previous major version to be removed in
> the
> > > next
> > > > > > major
> > > > > > >> >> > version.
> > > > > > >> >>
> > > > > > >> >> I totally agree that frequent major version bumps are not
> > > ideal,
> > > > > but
> > > > > > >> here
> > > > > > >> >> we are comparing it with a minor version bump which
> removes a
> > > > > Public
> > > > > > >> API.
> > > > > > >> >> So the context is that we have already decided to remove
> this
> > > > > Public
> > > > > > API
> > > > > > >> >> while keeping everything else backwards compatible. I
> think a
> > > > major
> > > > > > >> version
> > > > > > >> >> bump is a commonly understood signal here, compared with a
> > > minor
> > > > > > version
> > > > > > >> >> change. From end users' perspective, for those who are not
> > > > > impacted,
> > > > > > in
> > > > > > >> >> this case upgrading a major version is not necessarily more
> > > > > involved
> > > > > > >> than
> > > > > > >> >> upgrading a minor version - both should be as smooth as a
> > > > > dependency
> > > > > > >> >> version change. For those who are impacted, they will lose
> the
> > > > > Public
> > > > > > >> API
> > > > > > >> >> anyways and a major version bump ensures there is no
> surprise.
> > > > > > >> >>
> > > > > > >> >> Thanks,
> > > > > > >> >>
> > > > > > >> >> Jiangjie (Becket) Qin
> > > > > > >> >>
> > > > > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > tonysong820@gmail.com
> > > > > > >
> > > > > > >> >> wrote:
> > > > > > >> >>
> > > > > > >> >> > Public API is a well defined common concept, and one of
> its
> > > > > > >> >> >> convention is that it only changes with a major version
> > > > change.
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> > I agree. And from my understanding, demoting a Public
> API is
> > > > > also a
> > > > > > >> kind
> > > > > > >> >> > of such change, just like removing one, which can only
> happen
> > > > > with
> > > > > > >> major
> > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > > APIs
> > > > > > >> anytime,
> > > > > > >> >> but
> > > > > > >> >> > only in the case major version bumps happen before
> reaching
> > > the
> > > > > > >> >> > 2-minor-release migration period. Actually, demoting
> would
> > > be a
> > > > > > weaker
> > > > > > >> >> > change compared to removing the API immediately upon
> major
> > > > > version
> > > > > > >> bumps,
> > > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > > migration
> > > > > > >> >> period.
> > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving`
> sounds
> > > > > > against
> > > > > > >> >> > conventions, we may introduce a new annotation if
> necessary,
> > > > > e.g.,
> > > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > > >> >> >
> > > > > > >> >> > But it should be
> > > > > > >> >> >> completely OK to bump up the major version if we really
> want
> > > > to
> > > > > > get
> > > > > > >> rid
> > > > > > >> >> of
> > > > > > >> >> >> a public API, right?
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> > I'm not sure about this. Yes, it's completely "legal"
> that we
> > > > > bump
> > > > > > up
> > > > > > >> the
> > > > > > >> >> > major version whenever a breaking change is needed.
> However,
> > > > this
> > > > > > also
> > > > > > >> >> > weakens the value of the commitment that public APIs will
> > > stay
> > > > > > stable
> > > > > > >> >> > within the major release series, as the series can end
> > > anytime.
> > > > > > IMHO,
> > > > > > >> >> short
> > > > > > >> >> > major release series are not something "make the end
> users
> > > > > happy",
> > > > > > but
> > > > > > >> >> > backdoors that allow us as the developers to make
> frequent
> > > > > breaking
> > > > > > >> >> > changes. On the contrary, with the demoting approach, we
> can
> > > > > still
> > > > > > >> have
> > > > > > >> >> > longer major release series, while only allowing Public
> APIs
> > > > > > >> deprecated
> > > > > > >> >> at
> > > > > > >> >> > the end of the previous major version to be removed in
> the
> > > next
> > > > > > major
> > > > > > >> >> > version.
> > > > > > >> >> >
> > > > > > >> >> > Given our track record I would prefer a regular cycle
> (1-2
> > > > years)
> > > > > > to
> > > > > > >> >> >> force us to think about this whole topic, and not put it
> > > again
> > > > > to
> > > > > > the
> > > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > > when
> > > > > > >> breaking
> > > > > > >> >> >> changes can be made.
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> > +1. I personally think 2-3 years would be a good time
> for new
> > > > > major
> > > > > > >> >> > versions, or longer if there's no breaking changes
> needed.
> > > That
> > > > > > makes
> > > > > > >> 1-2
> > > > > > >> >> > year a perfect time to revisit the topic, while leaving
> us
> > > more
> > > > > > time
> > > > > > >> to
> > > > > > >> >> > prepare the major release if needed.
> > > > > > >> >> >
> > > > > > >> >> > Best,
> > > > > > >> >> >
> > > > > > >> >> > Xintong
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > > > chesnay@apache.org
> > > > > > >> >
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > >> >> >> > It would be valuable if we can avoid releasing minor
> > > > versions
> > > > > > for
> > > > > > >> >> >> previous
> > > > > > >> >> >> > major versions.
> > > > > > >> >> >>
> > > > > > >> >> >> On paper, /absolutely /agree, but I'm not sure how
> viable
> > > that
> > > > > is
> > > > > > in
> > > > > > >> >> >> practice.
> > > > > > >> >> >>
> > > > > > >> >> >> On the current 2.0 agenda is potentially dropping
> support
> > > for
> > > > > Java
> > > > > > >> 8/11,
> > > > > > >> >> >> which may very well be a problem for our current users.
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > >> >> >> > Thanks for the feedback and sorry for the confusion
> about
> > > > > Public
> > > > > > >> API
> > > > > > >> >> >> > deprecation. I just noticed that there was a mistake
> in
> > > the
> > > > > > NOTES
> > > > > > >> part
> > > > > > >> >> >> for
> > > > > > >> >> >> > Public API due to a copy-paste error... I just fixed
> it.
> > > > > > >> >> >> I'm very relieved to hear that. Glad to hear that we
> are on
> > > > the
> > > > > > same
> > > > > > >> >> >> page on that note.
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > >> >> >> > But it should be
> > > > > > >> >> >> > completely OK to bump up the major version if we
> really
> > > want
> > > > > to
> > > > > > get
> > > > > > >> >> rid
> > > > > > >> >> >> of
> > > > > > >> >> >> > a public API, right?
> > > > > > >> >> >>
> > > > > > >> >> >> Technically yes, but look at how long it took to get us
> to
> > > > 2.0.
> > > > > ;)
> > > > > > >> >> >>
> > > > > > >> >> >> There's a separate discussion to be had on the cadence
> of
> > > > major
> > > > > > >> releases
> > > > > > >> >> >> going forward, and there seem to be different opinions
> on
> > > > that.
> > > > > > >> >> >>
> > > > > > >> >> >> If we take the Kafka example of 2 minor releases between
> > > major
> > > > > > ones,
> > > > > > >> >> >> that for us means that users have to potentially deal
> with
> > > > > > breaking
> > > > > > >> >> >> changes every 6 months, which seems like a lot.
> > > > > > >> >> >>
> > > > > > >> >> >> Given our track record I would prefer a regular cycle
> (1-2
> > > > > years)
> > > > > > to
> > > > > > >> >> >> force us to think about this whole topic, and not put it
> > > again
> > > > > to
> > > > > > the
> > > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > > when
> > > > > > >> breaking
> > > > > > >> >> >> changes can be made.
> > > > > > >> >> >>
> > > > > > >> >> >> But again, maybe this should be in a separate thread.
> > > > > > >> >> >>
> > > > > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > >> >> >> > Do you have an example of behavioral change in mind?
> Not
> > > > sure
> > > > > I
> > > > > > >> fully
> > > > > > >> >> >> > understand the concern for behavioral change here.
> > > > > > >> >> >>
> > > > > > >> >> >> This could be a lot of things. It can be performance in
> > > > certain
> > > > > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly)
> relied
> > > > upon
> > > > > > >> >> >> (https://xkcd.com/1172/), a semantic change to some
> API.
> > > > > > >> >> >>
> > > > > > >> >> >> For a concrete example, consider the job submission. A
> few
> > > > > > releases
> > > > > > >> back
> > > > > > >> >> >> we made changes such that the initialization of the job
> > > master
> > > > > > >> happens
> > > > > > >> >> >> asynchronously.
> > > > > > >> >> >> This meant the job submission call returns sooner, and
> the
> > > job
> > > > > > state
> > > > > > >> >> >> enum was extended to cover this state.
> > > > > > >> >> >> API-wise we consider this a compatible change, but the
> > > > observed
> > > > > > >> behavior
> > > > > > >> >> >> may be different.
> > > > > > >> >> >>
> > > > > > >> >> >> Metrics are another example; I believe over time we
> changed
> > > > what
> > > > > > some
> > > > > > >> >> >> metrics returned a few times.
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by John Roesler <vv...@apache.org>.
Hi Becket and Xintong,

I hope you don't mind if I chime in a little.

Once an API is marked Public, we're committing to support it until it's deprecated, and once it's deprecated, to leave it in place for at least two minor releases and then only remove it in the following major release.

As a user, I would be dismayed to discover that the project maintainers demoted a Public API to PublicEvolving or Experimental in order to violate the spirit of the deprecation period for Public APIs. Likewise, I'd view a rapid sequence of minor releases for the purpose of dropping deprecated APIs in the next major release as another violation of the spirit of our guarantees. I'm reminded of this xkcd: https://xkcd.com/1494/

I'm glad we're gaming out these situations before we institute this FLIP, and I hope we all converge on a clear understanding of what we're guaranteeing. From where I'm sitting, the essence of this FLIP is specifically to prevent us from doing whatever we want and instead to commit to providing a suitable notice period to users who rely on Public APIs.

It honestly shouldn't matter what the maintenance overhead is (or, rather, evaluating the maintenance overhead should be a consideration before we promote an API to Public to begin with). My experience in other projects is that regretting a feature so much you want to drop it asap is extremely rare, and that living with the pain for a little while until the deprecation period expires will probably make us better engineers.

With that all said, one thing that is not a violation is to propose a new major release in order to drop especially burdensome deprecated APIs that have already enjoyed their full deprecation period. Those really bad "case A" features will be very rare.

Looking over the FLIP again, I see a reference to the release documentation, but not to the build tooling support for the Deprecated annotation, so it might help to share this scenario: The way this will all come together in practice is that, by giving good stability guarantees, we will encourage our users to bump minor versions when they are released, which means that they will see compiler warnings immediately if any of the APIs they've used are deprecated. In fact, I'd recommend for them to use the `-Werror` javac flag to be sure that they don't miss those warnings. Given our release cadence, this means that they will have about six months to migrate off of any deprecated APIs without being forced to avoid updates. And they really need to be able to adopt further minor updates in the series, since they may contain further improvements that actually facilitate migrating off of the deprecated APIs.

The compiler warnings are a far more reliable mechanism to advertise deprecation than release notes, since users' own builds will notify them right away about only the deprecations that are relevant to them, but it does rely on users feeling ok to bump up minor versions in a timely fashion. If we consistently burn them by not observing the stability guarantees, they'll try to avoid updates instead, and the whole thing falls apart.

Thanks again,
-John

On 2023/06/19 10:43:05 Becket Qin wrote:
> Hi Xintong,
> 
> Let's compare the following cases:
> 
> A. If the maintenance overhead of the deprecated API is high, and we want
> to remove it after two minor releases. Then there are two options:
>     A1: Demote the API in the major version bump and remove the code with a
> minor version bump.
>     A2: Do exactly the same as 1-A, except that when removing the code,
> bump up the major version. So this might be a short major version.
> 
> B. If the maintenance overhead of the deprecated API is not high, therefore
> keeping it for a long time is affordable. There are also two options:
>     B1: Same as A-1, demote the API in the major version bump and remove
> the code with a minor version bump.
>     B2: Keep the API for all the minor versions in the major version, and
> only remove the code in the next major version.
> 
> For case B, do we agree that B2 is the way to go?
> 
> For case A:
> The following stays the same for A1 and A2:
>   - users will lose the API after two minor leases. So the migration period
> is the same for A-1 and A-2.
>   - A1 and A2 will have exactly the same code after the removal of
> deprecated API, the only difference is versioning.
>   - To move forward, users need to move to the next minor version in A1, or
> to the next major version in A2. Because the code are the same, the actual
> effort to upgrade is the same for A1 and A2.
> 
> The differences between A-1 and A-2 are:
>   - A1 allows keeping the major version release cadence. A-2 will have a
> short major version release.
>   - A1 breaks the well understood API semantic, while A-2 does not.
> 
> From what I see, since there is no well establish standard regarding how
> long a major version should be released, A short major version release is
> potential and emotional. It is not ideal but does not have material
> downsides compared with A1. I did not hear anyone complaining about Kafka
> only has two 1.x release. However, A1 actually breaks the well understood
> API semantic, which has more material impact.
> 
> Also, I'd imagine 90% or more of the Public APIs should fall into case B.
> So, short major versions should be very occasional. I'd be very concerned
> if the reason we choose A1 is simply because we cannot afford maintaining a
> bunch of deprecated APIs until the next major version. This indicates that
> the actual problem we need to solve is to lower the maintenance overhead of
> deprecated APIs, so that we are comfortable to keep them longer. As John
> and I mentioned earlier, there are ways to achieve this and we need to
> learn how to do it in Flink. Otherwise, our discussion about versioning
> here does not bring much value, because we will end up with a bunch of
> short-lived APIs which upset our users, no matter how we version the
> releases.
> 
> So, if there are concrete examples that you think will block us from
> keeping API stability with affordable cost, let's take a look together and
> see if that can be improved.
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> 
> From what I see,
> 
> 
> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <to...@gmail.com> wrote:
> 
> > >
> > > The part I don't understand is if we are willing to have a migration
> > > period, and do a minor version bump to remove an API, what do we lose to
> > do
> > > a major version bump instead, so we don't break the common versioning
> > > semantic?
> > >
> >
> > I think we are talking about the cases where a major version bump happens
> > before a deprecated Public API reaches its migration period. So removing
> > the API with another major version bump means we have two consecutive major
> > versions in a very short time. And as previously mentioned, having frequent
> > major version bumps would weaken the value of the commitment "Public API
> > stay compatible within a major version". I think users also have
> > expectations about how long a major version should live, which should be at
> > least a couple of years rather than 1-2 minor releases.
> >
> > This is another option, but I think it is very likely more expensive than
> > > simply bumping the major version. And I imagine there will be questions
> > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > >
> >
> > Yes, it's more expensive than simply bumping the major version, but IMHO
> > it's probably cheaper than carrying the API until the next major version if
> > we don't bump the major version very soon. And see my reply above on why
> > not bumping immediately.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <be...@gmail.com> wrote:
> >
> > > Hi Xintong,
> > >
> > > Please see the replies below.
> > >
> > > I see your point, that users would feel surprised if they find things no
> > > > longer work when upgrading to another 2.x minor release. However, I'd
> > > like
> > > > to point out that PublicEvolving APIs would have the similar problem
> > > > anyway. So the question is, how do we catch users' attention and make
> > > sure
> > > > they are aware that the Public APIs in 1.x may no longer be Public in
> > > 2.0.
> > > > There're various ways to do that, e.g., release notes, warnings in
> > logs,
> > > > etc.
> > >
> > > First of all, I am not a fan of removing PublicEvolving APIs in minor
> > > version changes. Personally speaking, I want them to be also removed in
> > > major version changes.
> > >
> > > Empirically, I rarely see projects with complex API semantic work well.
> > > Many, if not most, users don't read docs / release notes / warnings in
> > > logs. I don't think that is their fault. Typically an application
> > developer
> > > will have to deal with dozens of libraries maintained by various
> > > communities / groups. It is just too difficult for them to keep track of
> > > all the specific semantics each project puts there. If you think about
> > it,
> > > how many of Flink developers read all the release notes of Guava, Apache
> > > Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of them?
> > > One would probably try bumping the dependency version and see if there is
> > > an exception and solve them case by case. And if it is a minor version
> > > bump, one probably does not expect an exception at all. Another example
> > is
> > > that in Flink we still have so many usage of deprecated methods all over
> > > the place, and I strongly doubt everyone knows when the source code of
> > > these methods are deprecated and when they should be removed.
> > >
> > > So, the most important principle of API is simple and intuitive. The
> > > versioning semantic is a simple and universally accepted API stability
> > > standard. If we ourselves as Flink developers are relying on this for our
> > > own dependencies. I don't think we can expect more from our users.
> > >
> > > Another possible alternative: whenever there's a deprecated Public API
> > that
> > > > reaches a major version bump before the migration period, and we also
> > > don't
> > > > want to carry it for all the next major release series, we may consider
> > > > releasing more minor releases for the previous major version after the
> > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to
> > 2.0,
> > > > we can release one more 1.20 after 2.0. That should provide users
> > another
> > > > choice rather than upgrading to 2.0, while satisfying the
> > 2-minor-release
> > > > migration period.
> > >
> > > This is another option, but I think it is very likely more expensive than
> > > simply bumping the major version. And I imagine there will be questions
> > > like "why this feature is in 1.20, but does not exist in 2.0"?
> > >
> > > I think my major point is, we should not carry APIs deprecated in a
> > > > previous major version along all the next major version series. I'd
> > like
> > > to
> > > > try giving users more commitments, i.e. the migration period, as long
> > as
> > > it
> > > > does not prevent us from making breaking changes. If it doesn't work,
> > I'd
> > > > be in favor of not providing the migration period, but fallback to only
> > > > guarantee the compatibility within the major version.
> > >
> > > The part I don't understand is if we are willing to have a migration
> > > period, and do a minor version bump to remove an API, what do we lose to
> > do
> > > a major version bump instead, so we don't break the common versioning
> > > semantic?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > > On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com>
> > > wrote:
> > >
> > > > >
> > > > > As an end user who only uses Public APIs, if I don't change my code
> > at
> > > > > all, my expectation is the following:
> > > > > 1. Upgrading from 1.x to 2.x may have issues.
> > > > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with
> > > all
> > > > > the 2.x versions.
> > > > > Actually I think there are some dependency version resolution
> > policies
> > > > out
> > > > > there which picks the highest minor version when the dependencies
> > pull
> > > in
> > > > > multiple minor versions of the same jar, which may be broken if we
> > > remove
> > > > > the API in minor releases.
> > > > >
> > > >
> > > > I see your point, that users would feel surprised if they find things
> > no
> > > > longer work when upgrading to another 2.x minor release. However, I'd
> > > like
> > > > to point out that PublicEvolving APIs would have the similar problem
> > > > anyway. So the question is, how do we catch users' attention and make
> > > sure
> > > > they are aware that the Public APIs in 1.x may no longer be Public in
> > > 2.0.
> > > > There're various ways to do that, e.g., release notes, warnings in
> > logs,
> > > > etc.
> > > >
> > > > Another possible alternative: whenever there's a deprecated Public API
> > > that
> > > > reaches a major version bump before the migration period, and we also
> > > don't
> > > > want to carry it for all the next major release series, we may consider
> > > > releasing more minor releases for the previous major version after the
> > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to
> > 2.0,
> > > > we can release one more 1.20 after 2.0. That should provide users
> > another
> > > > choice rather than upgrading to 2.0, while satisfying the
> > 2-minor-release
> > > > migration period.
> > > >
> > > > I think my major point is, we should not carry APIs deprecated in a
> > > > previous major version along all the next major version series. I'd
> > like
> > > to
> > > > try giving users more commitments, i.e. the migration period, as long
> > as
> > > it
> > > > does not prevent us from making breaking changes. If it doesn't work,
> > I'd
> > > > be in favor of not providing the migration period, but fallback to only
> > > > guarantee the compatibility within the major version.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Becket,
> > > > >
> > > > > Thanks for the reply! I’d like to continue the conversation about
> > > > > compatibility outside of this FLIP thread, but for now, I can accept
> > > your
> > > > > decision. It’s certainly an improvement.
> > > > >
> > > > > Thanks again,
> > > > > John
> > > > >
> > > > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > Hi John,
> > > > > >
> > > > > > Completely agree with all you said.
> > > > > >
> > > > > > Can we consider only dropping deprecated APIs in major releases
> > > across
> > > > > the
> > > > > >> board? I understand that Experimental and PublicEvolving APIs are
> > by
> > > > > >> definition less stable, but it seems like this should be reflected
> > > in
> > > > > the
> > > > > >> required deprecation period alone. I.e. that we must keep them
> > > around
> > > > > for
> > > > > >> at least zero or one minor release, not that we can drop them in a
> > > > > minor or
> > > > > >> patch release.
> > > > > >
> > > > > > Personally speaking, I would love to do this, for exactly the
> > reason
> > > > you
> > > > > > mentioned. However, I did not propose this due to the following
> > > > reasons:
> > > > > >
> > > > > > 1. I am hesitating a little bit about changing the accepted FLIPs
> > too
> > > > > soon.
> > > > > > 2. More importantly, to avoid slowing down our development. At this
> > > > > point,
> > > > > > Flink still lacks some design / routines to support good API
> > > > > evolvability /
> > > > > > extensibility. Just like you said, it takes some time to be good at
> > > > this.
> > > > > > In this case, my concern is that only removing Experimental /
> > > > > > PublicEvolving APIs in major version changes may result in too much
> > > > > > overhead and dramatically slow down the development of Flink. So, I
> > > was
> > > > > > thinking that we can start with the current status. Hopefully after
> > > we
> > > > > are
> > > > > > more comfortable with the maintenance overhead of deprecated APIs,
> > we
> > > > can
> > > > > > then have a stronger guarantee for Experimental / PublicEvolving
> > > APIs.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Becket,
> > > > > >>
> > > > > >> Thanks for this FLIP! Having a deprecation process is really
> > > > important.
> > > > > I
> > > > > >> understand some people’s concerns about the additional burden for
> > > > > project
> > > > > >> maintainers, but my personal experience with Kafka has been that
> > > it’s
> > > > > very
> > > > > >> liveable and that it’s well worth the benefit to users. In fact,
> > > users
> > > > > >> being able to confidently upgrade is also a benefit to
> > maintainers,
> > > as
> > > > > we
> > > > > >> will get fewer questions from people stuck on very old versions.
> > > > > >>
> > > > > >> One question:
> > > > > >> Can we consider only dropping deprecated APIs in major releases
> > > across
> > > > > the
> > > > > >> board? I understand that Experimental and PublicEvolving APIs are
> > by
> > > > > >> definition less stable, but it seems like this should be reflected
> > > in
> > > > > the
> > > > > >> required deprecation period alone. I.e. that we must keep them
> > > around
> > > > > for
> > > > > >> at least zero or one minor release, not that we can drop them in a
> > > > > minor or
> > > > > >> patch release.
> > > > > >>
> > > > > >> The advantage of forbidding the removal of any API in minor or
> > patch
> > > > > >> releases is that users will get a strong guarantee that they can
> > > bump
> > > > > the
> > > > > >> minor or patch version and still be able to compile, or even just
> > > > > re-link
> > > > > >> and know that they won’t face “MethodDef” exceptions at run time.
> > > This
> > > > > is a
> > > > > >> binary guarantee: if we allow removing  even Experimental APIs
> > > outside
> > > > > of
> > > > > >> major releases, users can no longer confidently upgrade.
> > > > > >>
> > > > > >> Aside from that, I’d share my 2 cents on a couple of points:
> > > > > >> * I’d use the official Deprecated annotation instead of
> > introducing
> > > > our
> > > > > >> own flavor (Retired, etc), since Deprecated is well integrated
> > into
> > > > > build
> > > > > >> tools and IDEs.
> > > > > >> * I wouldn’t worry about a demotion process in this FLIP; it seems
> > > > > >> orthogonal, and something that should probably be taken
> > case-by-case
> > > > > >> anyway.
> > > > > >> * Aside from deprecation and removal, there have been some
> > > discussions
> > > > > >> about how to evolve APIs and behavior in compatible ways. This is
> > > > > somewhat
> > > > > >> of an art, and if folks haven’t wrestled with it before, it’ll
> > take
> > > > some
> > > > > >> time to become good at it. I feel like this topic should also be
> > > > > orthogonal
> > > > > >> to this FLIP, but FWIW, my suggestion would be to adopt a simple
> > > > policy
> > > > > not
> > > > > >> to break existing user programs, and leave the “how” up to
> > > > implementers
> > > > > and
> > > > > >> reviewers.
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> John
> > > > > >>
> > > > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > The @Public -> @PublicEvolving proposed by Xintong is a great
> > > idea.
> > > > > >> > Especially, after he suggest @PublicRetired, i.e.
> > @PublicEvolving
> > > > --(2
> > > > > >> > minor release)--> @Public --> @deprecated --(1 major
> > > > > >> > release)--> @PublicRetired. It will provide a lot of flexibility
> > > > > without
> > > > > >> > breaking any rules we had. @Public APIs are allowed to change
> > > > between
> > > > > >> major
> > > > > >> > releases. Changing annotations is acceptable and provides
> > > additional
> > > > > >> > tolerance i.e. user-friendliness, since the APIs themself are
> > not
> > > > > >> changed.
> > > > > >> >
> > > > > >> > I had similar thoughts when I was facing those issues. I want to
> > > > move
> > > > > one
> > > > > >> > step further and suggest introducing one more annotation
> > @Retired.
> > > > > >> >
> > > > > >> > Not like the @PublicRetired which is a compromise of downgrading
> > > > > @Public
> > > > > >> to
> > > > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java
> > standard
> > > > > >> > @deprecated should be used in the early stage of the deprecation
> > > > > process
> > > > > >> > and doesn't really meet our requirement. Since Java does not
> > allow
> > > > us
> > > > > to
> > > > > >> > extend annotation, I think it would be feasible to have the new
> > > > > @Retired
> > > > > >> to
> > > > > >> > help us monitor and manage the deprecation process, house
> > > cleaning,
> > > > > etc.
> > > > > >> >
> > > > > >> > Some ideas could be(open for discussion):
> > > > > >> >
> > > > > >> > @Retired:
> > > > > >> >
> > > > > >> > 1. There must be a replacement with functionality compatibility
> > > > before
> > > > > >> APIs
> > > > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on
> > the
> > > ML
> > > > > are
> > > > > >> > mandatory (a FLIP is recommended).
> > > > > >> > 2. APIs marked as @Retired will be removed after 1 minor release
> > > > > sharply
> > > > > >> > (using ArchUnit to force it, needs further check whether it is
> > > > > possible).
> > > > > >> > Devs who marked them as @Retired are responsible to remove them.
> > > > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> > > > > >> > recommended. @Experimental -> @Retired and @Internal -> @Retired
> > > > could
> > > > > >> also
> > > > > >> > be used if it can increase user-friendliness or
> > dev-friendliness,
> > > > but
> > > > > not
> > > > > >> > mandatory.
> > > > > >> > 4. Some variables will be defined in @Retired to support the
> > > > > deprecation
> > > > > >> > process management. Further extension is possible, since the
> > > > > annotation
> > > > > >> is
> > > > > >> > built by us.
> > > > > >> >
> > > > > >> >
> > > > > >> > Best regards,
> > > > > >> > Jing
> > > > > >> >
> > > > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > becket.qin@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> >> Hi Xintong,
> > > > > >> >>
> > > > > >> >> Thanks for the explanation. Please see the replies inline
> > below.
> > > > > >> >>
> > > > > >> >> I agree. And from my understanding, demoting a Public API is
> > > also a
> > > > > >> kind of
> > > > > >> >> > such change, just like removing one, which can only happen
> > with
> > > > > major
> > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > APIs
> > > > > >> anytime,
> > > > > >> >> but
> > > > > >> >> > only in the case major version bumps happen before reaching
> > the
> > > > > >> >> > 2-minor-release migration period. Actually, demoting would
> > be a
> > > > > weaker
> > > > > >> >> > change compared to removing the API immediately upon major
> > > > version
> > > > > >> bumps,
> > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > migration
> > > > > >> >> period.
> > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > > > against
> > > > > >> >> > conventions, we may introduce a new annotation if necessary,
> > > > e.g.,
> > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > >> >>
> > > > > >> >> As an end user who only uses Public APIs, if I don't change my
> > > code
> > > > > at
> > > > > >> all,
> > > > > >> >> my expectation is the following:
> > > > > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > > > with
> > > > > all
> > > > > >> >> the 2.x versions.
> > > > > >> >> Actually I think there are some dependency version resolution
> > > > > policies
> > > > > >> out
> > > > > >> >> there which picks the highest minor version when the
> > dependencies
> > > > > pull
> > > > > >> in
> > > > > >> >> multiple minor versions of the same jar, which may be broken if
> > > we
> > > > > >> remove
> > > > > >> >> the API in minor releases.
> > > > > >> >>
> > > > > >> >> I'm not sure about this. Yes, it's completely "legal" that we
> > > bump
> > > > up
> > > > > >> the
> > > > > >> >> > major version whenever a breaking change is needed. However,
> > > this
> > > > > also
> > > > > >> >> > weakens the value of the commitment that public APIs will
> > stay
> > > > > stable
> > > > > >> >> > within the major release series, as the series can end
> > anytime.
> > > > > IMHO,
> > > > > >> >> short
> > > > > >> >> > major release series are not something "make the end users
> > > > happy",
> > > > > but
> > > > > >> >> > backdoors that allow us as the developers to make frequent
> > > > breaking
> > > > > >> >> > changes. On the contrary, with the demoting approach, we can
> > > > still
> > > > > >> have
> > > > > >> >> > longer major release series, while only allowing Public APIs
> > > > > >> deprecated
> > > > > >> >> at
> > > > > >> >> > the end of the previous major version to be removed in the
> > next
> > > > > major
> > > > > >> >> > version.
> > > > > >> >>
> > > > > >> >> I totally agree that frequent major version bumps are not
> > ideal,
> > > > but
> > > > > >> here
> > > > > >> >> we are comparing it with a minor version bump which removes a
> > > > Public
> > > > > >> API.
> > > > > >> >> So the context is that we have already decided to remove this
> > > > Public
> > > > > API
> > > > > >> >> while keeping everything else backwards compatible. I think a
> > > major
> > > > > >> version
> > > > > >> >> bump is a commonly understood signal here, compared with a
> > minor
> > > > > version
> > > > > >> >> change. From end users' perspective, for those who are not
> > > > impacted,
> > > > > in
> > > > > >> >> this case upgrading a major version is not necessarily more
> > > > involved
> > > > > >> than
> > > > > >> >> upgrading a minor version - both should be as smooth as a
> > > > dependency
> > > > > >> >> version change. For those who are impacted, they will lose the
> > > > Public
> > > > > >> API
> > > > > >> >> anyways and a major version bump ensures there is no surprise.
> > > > > >> >>
> > > > > >> >> Thanks,
> > > > > >> >>
> > > > > >> >> Jiangjie (Becket) Qin
> > > > > >> >>
> > > > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > tonysong820@gmail.com
> > > > > >
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Public API is a well defined common concept, and one of its
> > > > > >> >> >> convention is that it only changes with a major version
> > > change.
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> > I agree. And from my understanding, demoting a Public API is
> > > > also a
> > > > > >> kind
> > > > > >> >> > of such change, just like removing one, which can only happen
> > > > with
> > > > > >> major
> > > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> > APIs
> > > > > >> anytime,
> > > > > >> >> but
> > > > > >> >> > only in the case major version bumps happen before reaching
> > the
> > > > > >> >> > 2-minor-release migration period. Actually, demoting would
> > be a
> > > > > weaker
> > > > > >> >> > change compared to removing the API immediately upon major
> > > > version
> > > > > >> bumps,
> > > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > > migration
> > > > > >> >> period.
> > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > > > against
> > > > > >> >> > conventions, we may introduce a new annotation if necessary,
> > > > e.g.,
> > > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > > >> >> >
> > > > > >> >> > But it should be
> > > > > >> >> >> completely OK to bump up the major version if we really want
> > > to
> > > > > get
> > > > > >> rid
> > > > > >> >> of
> > > > > >> >> >> a public API, right?
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> > I'm not sure about this. Yes, it's completely "legal" that we
> > > > bump
> > > > > up
> > > > > >> the
> > > > > >> >> > major version whenever a breaking change is needed. However,
> > > this
> > > > > also
> > > > > >> >> > weakens the value of the commitment that public APIs will
> > stay
> > > > > stable
> > > > > >> >> > within the major release series, as the series can end
> > anytime.
> > > > > IMHO,
> > > > > >> >> short
> > > > > >> >> > major release series are not something "make the end users
> > > > happy",
> > > > > but
> > > > > >> >> > backdoors that allow us as the developers to make frequent
> > > > breaking
> > > > > >> >> > changes. On the contrary, with the demoting approach, we can
> > > > still
> > > > > >> have
> > > > > >> >> > longer major release series, while only allowing Public APIs
> > > > > >> deprecated
> > > > > >> >> at
> > > > > >> >> > the end of the previous major version to be removed in the
> > next
> > > > > major
> > > > > >> >> > version.
> > > > > >> >> >
> > > > > >> >> > Given our track record I would prefer a regular cycle (1-2
> > > years)
> > > > > to
> > > > > >> >> >> force us to think about this whole topic, and not put it
> > again
> > > > to
> > > > > the
> > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > when
> > > > > >> breaking
> > > > > >> >> >> changes can be made.
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> > +1. I personally think 2-3 years would be a good time for new
> > > > major
> > > > > >> >> > versions, or longer if there's no breaking changes needed.
> > That
> > > > > makes
> > > > > >> 1-2
> > > > > >> >> > year a perfect time to revisit the topic, while leaving us
> > more
> > > > > time
> > > > > >> to
> > > > > >> >> > prepare the major release if needed.
> > > > > >> >> >
> > > > > >> >> > Best,
> > > > > >> >> >
> > > > > >> >> > Xintong
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > > chesnay@apache.org
> > > > > >> >
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > >> >> >> > It would be valuable if we can avoid releasing minor
> > > versions
> > > > > for
> > > > > >> >> >> previous
> > > > > >> >> >> > major versions.
> > > > > >> >> >>
> > > > > >> >> >> On paper, /absolutely /agree, but I'm not sure how viable
> > that
> > > > is
> > > > > in
> > > > > >> >> >> practice.
> > > > > >> >> >>
> > > > > >> >> >> On the current 2.0 agenda is potentially dropping support
> > for
> > > > Java
> > > > > >> 8/11,
> > > > > >> >> >> which may very well be a problem for our current users.
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > >> >> >> > Thanks for the feedback and sorry for the confusion about
> > > > Public
> > > > > >> API
> > > > > >> >> >> > deprecation. I just noticed that there was a mistake in
> > the
> > > > > NOTES
> > > > > >> part
> > > > > >> >> >> for
> > > > > >> >> >> > Public API due to a copy-paste error... I just fixed it.
> > > > > >> >> >> I'm very relieved to hear that. Glad to hear that we are on
> > > the
> > > > > same
> > > > > >> >> >> page on that note.
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > >> >> >> > But it should be
> > > > > >> >> >> > completely OK to bump up the major version if we really
> > want
> > > > to
> > > > > get
> > > > > >> >> rid
> > > > > >> >> >> of
> > > > > >> >> >> > a public API, right?
> > > > > >> >> >>
> > > > > >> >> >> Technically yes, but look at how long it took to get us to
> > > 2.0.
> > > > ;)
> > > > > >> >> >>
> > > > > >> >> >> There's a separate discussion to be had on the cadence of
> > > major
> > > > > >> releases
> > > > > >> >> >> going forward, and there seem to be different opinions on
> > > that.
> > > > > >> >> >>
> > > > > >> >> >> If we take the Kafka example of 2 minor releases between
> > major
> > > > > ones,
> > > > > >> >> >> that for us means that users have to potentially deal with
> > > > > breaking
> > > > > >> >> >> changes every 6 months, which seems like a lot.
> > > > > >> >> >>
> > > > > >> >> >> Given our track record I would prefer a regular cycle (1-2
> > > > years)
> > > > > to
> > > > > >> >> >> force us to think about this whole topic, and not put it
> > again
> > > > to
> > > > > the
> > > > > >> >> >> wayside and giving us (and users) a clear expectation on
> > when
> > > > > >> breaking
> > > > > >> >> >> changes can be made.
> > > > > >> >> >>
> > > > > >> >> >> But again, maybe this should be in a separate thread.
> > > > > >> >> >>
> > > > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > >> >> >> > Do you have an example of behavioral change in mind? Not
> > > sure
> > > > I
> > > > > >> fully
> > > > > >> >> >> > understand the concern for behavioral change here.
> > > > > >> >> >>
> > > > > >> >> >> This could be a lot of things. It can be performance in
> > > certain
> > > > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied
> > > upon
> > > > > >> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> > > > > >> >> >>
> > > > > >> >> >> For a concrete example, consider the job submission. A few
> > > > > releases
> > > > > >> back
> > > > > >> >> >> we made changes such that the initialization of the job
> > master
> > > > > >> happens
> > > > > >> >> >> asynchronously.
> > > > > >> >> >> This meant the job submission call returns sooner, and the
> > job
> > > > > state
> > > > > >> >> >> enum was extended to cover this state.
> > > > > >> >> >> API-wise we consider this a compatible change, but the
> > > observed
> > > > > >> behavior
> > > > > >> >> >> may be different.
> > > > > >> >> >>
> > > > > >> >> >> Metrics are another example; I believe over time we changed
> > > what
> > > > > some
> > > > > >> >> >> metrics returned a few times.
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

Let's compare the following cases:

A. If the maintenance overhead of the deprecated API is high, and we want
to remove it after two minor releases. Then there are two options:
    A1: Demote the API in the major version bump and remove the code with a
minor version bump.
    A2: Do exactly the same as 1-A, except that when removing the code,
bump up the major version. So this might be a short major version.

B. If the maintenance overhead of the deprecated API is not high, therefore
keeping it for a long time is affordable. There are also two options:
    B1: Same as A-1, demote the API in the major version bump and remove
the code with a minor version bump.
    B2: Keep the API for all the minor versions in the major version, and
only remove the code in the next major version.

For case B, do we agree that B2 is the way to go?

For case A:
The following stays the same for A1 and A2:
  - users will lose the API after two minor leases. So the migration period
is the same for A-1 and A-2.
  - A1 and A2 will have exactly the same code after the removal of
deprecated API, the only difference is versioning.
  - To move forward, users need to move to the next minor version in A1, or
to the next major version in A2. Because the code are the same, the actual
effort to upgrade is the same for A1 and A2.

The differences between A-1 and A-2 are:
  - A1 allows keeping the major version release cadence. A-2 will have a
short major version release.
  - A1 breaks the well understood API semantic, while A-2 does not.

From what I see, since there is no well establish standard regarding how
long a major version should be released, A short major version release is
potential and emotional. It is not ideal but does not have material
downsides compared with A1. I did not hear anyone complaining about Kafka
only has two 1.x release. However, A1 actually breaks the well understood
API semantic, which has more material impact.

Also, I'd imagine 90% or more of the Public APIs should fall into case B.
So, short major versions should be very occasional. I'd be very concerned
if the reason we choose A1 is simply because we cannot afford maintaining a
bunch of deprecated APIs until the next major version. This indicates that
the actual problem we need to solve is to lower the maintenance overhead of
deprecated APIs, so that we are comfortable to keep them longer. As John
and I mentioned earlier, there are ways to achieve this and we need to
learn how to do it in Flink. Otherwise, our discussion about versioning
here does not bring much value, because we will end up with a bunch of
short-lived APIs which upset our users, no matter how we version the
releases.

So, if there are concrete examples that you think will block us from
keeping API stability with affordable cost, let's take a look together and
see if that can be improved.

Thanks,

Jiangjie (Becket) Qin


From what I see,


On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <to...@gmail.com> wrote:

> >
> > The part I don't understand is if we are willing to have a migration
> > period, and do a minor version bump to remove an API, what do we lose to
> do
> > a major version bump instead, so we don't break the common versioning
> > semantic?
> >
>
> I think we are talking about the cases where a major version bump happens
> before a deprecated Public API reaches its migration period. So removing
> the API with another major version bump means we have two consecutive major
> versions in a very short time. And as previously mentioned, having frequent
> major version bumps would weaken the value of the commitment "Public API
> stay compatible within a major version". I think users also have
> expectations about how long a major version should live, which should be at
> least a couple of years rather than 1-2 minor releases.
>
> This is another option, but I think it is very likely more expensive than
> > simply bumping the major version. And I imagine there will be questions
> > like "why this feature is in 1.20, but does not exist in 2.0"?
> >
>
> Yes, it's more expensive than simply bumping the major version, but IMHO
> it's probably cheaper than carrying the API until the next major version if
> we don't bump the major version very soon. And see my reply above on why
> not bumping immediately.
>
> Best,
>
> Xintong
>
>
>
> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <be...@gmail.com> wrote:
>
> > Hi Xintong,
> >
> > Please see the replies below.
> >
> > I see your point, that users would feel surprised if they find things no
> > > longer work when upgrading to another 2.x minor release. However, I'd
> > like
> > > to point out that PublicEvolving APIs would have the similar problem
> > > anyway. So the question is, how do we catch users' attention and make
> > sure
> > > they are aware that the Public APIs in 1.x may no longer be Public in
> > 2.0.
> > > There're various ways to do that, e.g., release notes, warnings in
> logs,
> > > etc.
> >
> > First of all, I am not a fan of removing PublicEvolving APIs in minor
> > version changes. Personally speaking, I want them to be also removed in
> > major version changes.
> >
> > Empirically, I rarely see projects with complex API semantic work well.
> > Many, if not most, users don't read docs / release notes / warnings in
> > logs. I don't think that is their fault. Typically an application
> developer
> > will have to deal with dozens of libraries maintained by various
> > communities / groups. It is just too difficult for them to keep track of
> > all the specific semantics each project puts there. If you think about
> it,
> > how many of Flink developers read all the release notes of Guava, Apache
> > Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of them?
> > One would probably try bumping the dependency version and see if there is
> > an exception and solve them case by case. And if it is a minor version
> > bump, one probably does not expect an exception at all. Another example
> is
> > that in Flink we still have so many usage of deprecated methods all over
> > the place, and I strongly doubt everyone knows when the source code of
> > these methods are deprecated and when they should be removed.
> >
> > So, the most important principle of API is simple and intuitive. The
> > versioning semantic is a simple and universally accepted API stability
> > standard. If we ourselves as Flink developers are relying on this for our
> > own dependencies. I don't think we can expect more from our users.
> >
> > Another possible alternative: whenever there's a deprecated Public API
> that
> > > reaches a major version bump before the migration period, and we also
> > don't
> > > want to carry it for all the next major release series, we may consider
> > > releasing more minor releases for the previous major version after the
> > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to
> 2.0,
> > > we can release one more 1.20 after 2.0. That should provide users
> another
> > > choice rather than upgrading to 2.0, while satisfying the
> 2-minor-release
> > > migration period.
> >
> > This is another option, but I think it is very likely more expensive than
> > simply bumping the major version. And I imagine there will be questions
> > like "why this feature is in 1.20, but does not exist in 2.0"?
> >
> > I think my major point is, we should not carry APIs deprecated in a
> > > previous major version along all the next major version series. I'd
> like
> > to
> > > try giving users more commitments, i.e. the migration period, as long
> as
> > it
> > > does not prevent us from making breaking changes. If it doesn't work,
> I'd
> > > be in favor of not providing the migration period, but fallback to only
> > > guarantee the compatibility within the major version.
> >
> > The part I don't understand is if we are willing to have a migration
> > period, and do a minor version bump to remove an API, what do we lose to
> do
> > a major version bump instead, so we don't break the common versioning
> > semantic?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > >
> > > > As an end user who only uses Public APIs, if I don't change my code
> at
> > > > all, my expectation is the following:
> > > > 1. Upgrading from 1.x to 2.x may have issues.
> > > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with
> > all
> > > > the 2.x versions.
> > > > Actually I think there are some dependency version resolution
> policies
> > > out
> > > > there which picks the highest minor version when the dependencies
> pull
> > in
> > > > multiple minor versions of the same jar, which may be broken if we
> > remove
> > > > the API in minor releases.
> > > >
> > >
> > > I see your point, that users would feel surprised if they find things
> no
> > > longer work when upgrading to another 2.x minor release. However, I'd
> > like
> > > to point out that PublicEvolving APIs would have the similar problem
> > > anyway. So the question is, how do we catch users' attention and make
> > sure
> > > they are aware that the Public APIs in 1.x may no longer be Public in
> > 2.0.
> > > There're various ways to do that, e.g., release notes, warnings in
> logs,
> > > etc.
> > >
> > > Another possible alternative: whenever there's a deprecated Public API
> > that
> > > reaches a major version bump before the migration period, and we also
> > don't
> > > want to carry it for all the next major release series, we may consider
> > > releasing more minor releases for the previous major version after the
> > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to
> 2.0,
> > > we can release one more 1.20 after 2.0. That should provide users
> another
> > > choice rather than upgrading to 2.0, while satisfying the
> 2-minor-release
> > > migration period.
> > >
> > > I think my major point is, we should not carry APIs deprecated in a
> > > previous major version along all the next major version series. I'd
> like
> > to
> > > try giving users more commitments, i.e. the migration period, as long
> as
> > it
> > > does not prevent us from making breaking changes. If it doesn't work,
> I'd
> > > be in favor of not providing the migration period, but fallback to only
> > > guarantee the compatibility within the major version.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > > > Hi Becket,
> > > >
> > > > Thanks for the reply! I’d like to continue the conversation about
> > > > compatibility outside of this FLIP thread, but for now, I can accept
> > your
> > > > decision. It’s certainly an improvement.
> > > >
> > > > Thanks again,
> > > > John
> > > >
> > > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > Hi John,
> > > > >
> > > > > Completely agree with all you said.
> > > > >
> > > > > Can we consider only dropping deprecated APIs in major releases
> > across
> > > > the
> > > > >> board? I understand that Experimental and PublicEvolving APIs are
> by
> > > > >> definition less stable, but it seems like this should be reflected
> > in
> > > > the
> > > > >> required deprecation period alone. I.e. that we must keep them
> > around
> > > > for
> > > > >> at least zero or one minor release, not that we can drop them in a
> > > > minor or
> > > > >> patch release.
> > > > >
> > > > > Personally speaking, I would love to do this, for exactly the
> reason
> > > you
> > > > > mentioned. However, I did not propose this due to the following
> > > reasons:
> > > > >
> > > > > 1. I am hesitating a little bit about changing the accepted FLIPs
> too
> > > > soon.
> > > > > 2. More importantly, to avoid slowing down our development. At this
> > > > point,
> > > > > Flink still lacks some design / routines to support good API
> > > > evolvability /
> > > > > extensibility. Just like you said, it takes some time to be good at
> > > this.
> > > > > In this case, my concern is that only removing Experimental /
> > > > > PublicEvolving APIs in major version changes may result in too much
> > > > > overhead and dramatically slow down the development of Flink. So, I
> > was
> > > > > thinking that we can start with the current status. Hopefully after
> > we
> > > > are
> > > > > more comfortable with the maintenance overhead of deprecated APIs,
> we
> > > can
> > > > > then have a stronger guarantee for Experimental / PublicEvolving
> > APIs.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > >> Hi Becket,
> > > > >>
> > > > >> Thanks for this FLIP! Having a deprecation process is really
> > > important.
> > > > I
> > > > >> understand some people’s concerns about the additional burden for
> > > > project
> > > > >> maintainers, but my personal experience with Kafka has been that
> > it’s
> > > > very
> > > > >> liveable and that it’s well worth the benefit to users. In fact,
> > users
> > > > >> being able to confidently upgrade is also a benefit to
> maintainers,
> > as
> > > > we
> > > > >> will get fewer questions from people stuck on very old versions.
> > > > >>
> > > > >> One question:
> > > > >> Can we consider only dropping deprecated APIs in major releases
> > across
> > > > the
> > > > >> board? I understand that Experimental and PublicEvolving APIs are
> by
> > > > >> definition less stable, but it seems like this should be reflected
> > in
> > > > the
> > > > >> required deprecation period alone. I.e. that we must keep them
> > around
> > > > for
> > > > >> at least zero or one minor release, not that we can drop them in a
> > > > minor or
> > > > >> patch release.
> > > > >>
> > > > >> The advantage of forbidding the removal of any API in minor or
> patch
> > > > >> releases is that users will get a strong guarantee that they can
> > bump
> > > > the
> > > > >> minor or patch version and still be able to compile, or even just
> > > > re-link
> > > > >> and know that they won’t face “MethodDef” exceptions at run time.
> > This
> > > > is a
> > > > >> binary guarantee: if we allow removing  even Experimental APIs
> > outside
> > > > of
> > > > >> major releases, users can no longer confidently upgrade.
> > > > >>
> > > > >> Aside from that, I’d share my 2 cents on a couple of points:
> > > > >> * I’d use the official Deprecated annotation instead of
> introducing
> > > our
> > > > >> own flavor (Retired, etc), since Deprecated is well integrated
> into
> > > > build
> > > > >> tools and IDEs.
> > > > >> * I wouldn’t worry about a demotion process in this FLIP; it seems
> > > > >> orthogonal, and something that should probably be taken
> case-by-case
> > > > >> anyway.
> > > > >> * Aside from deprecation and removal, there have been some
> > discussions
> > > > >> about how to evolve APIs and behavior in compatible ways. This is
> > > > somewhat
> > > > >> of an art, and if folks haven’t wrestled with it before, it’ll
> take
> > > some
> > > > >> time to become good at it. I feel like this topic should also be
> > > > orthogonal
> > > > >> to this FLIP, but FWIW, my suggestion would be to adopt a simple
> > > policy
> > > > not
> > > > >> to break existing user programs, and leave the “how” up to
> > > implementers
> > > > and
> > > > >> reviewers.
> > > > >>
> > > > >> Thanks again,
> > > > >> John
> > > > >>
> > > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > >> > Hi All,
> > > > >> >
> > > > >> > The @Public -> @PublicEvolving proposed by Xintong is a great
> > idea.
> > > > >> > Especially, after he suggest @PublicRetired, i.e.
> @PublicEvolving
> > > --(2
> > > > >> > minor release)--> @Public --> @deprecated --(1 major
> > > > >> > release)--> @PublicRetired. It will provide a lot of flexibility
> > > > without
> > > > >> > breaking any rules we had. @Public APIs are allowed to change
> > > between
> > > > >> major
> > > > >> > releases. Changing annotations is acceptable and provides
> > additional
> > > > >> > tolerance i.e. user-friendliness, since the APIs themself are
> not
> > > > >> changed.
> > > > >> >
> > > > >> > I had similar thoughts when I was facing those issues. I want to
> > > move
> > > > one
> > > > >> > step further and suggest introducing one more annotation
> @Retired.
> > > > >> >
> > > > >> > Not like the @PublicRetired which is a compromise of downgrading
> > > > @Public
> > > > >> to
> > > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java
> standard
> > > > >> > @deprecated should be used in the early stage of the deprecation
> > > > process
> > > > >> > and doesn't really meet our requirement. Since Java does not
> allow
> > > us
> > > > to
> > > > >> > extend annotation, I think it would be feasible to have the new
> > > > @Retired
> > > > >> to
> > > > >> > help us monitor and manage the deprecation process, house
> > cleaning,
> > > > etc.
> > > > >> >
> > > > >> > Some ideas could be(open for discussion):
> > > > >> >
> > > > >> > @Retired:
> > > > >> >
> > > > >> > 1. There must be a replacement with functionality compatibility
> > > before
> > > > >> APIs
> > > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on
> the
> > ML
> > > > are
> > > > >> > mandatory (a FLIP is recommended).
> > > > >> > 2. APIs marked as @Retired will be removed after 1 minor release
> > > > sharply
> > > > >> > (using ArchUnit to force it, needs further check whether it is
> > > > possible).
> > > > >> > Devs who marked them as @Retired are responsible to remove them.
> > > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> > > > >> > recommended. @Experimental -> @Retired and @Internal -> @Retired
> > > could
> > > > >> also
> > > > >> > be used if it can increase user-friendliness or
> dev-friendliness,
> > > but
> > > > not
> > > > >> > mandatory.
> > > > >> > 4. Some variables will be defined in @Retired to support the
> > > > deprecation
> > > > >> > process management. Further extension is possible, since the
> > > > annotation
> > > > >> is
> > > > >> > built by us.
> > > > >> >
> > > > >> >
> > > > >> > Best regards,
> > > > >> > Jing
> > > > >> >
> > > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> becket.qin@gmail.com
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> >> Hi Xintong,
> > > > >> >>
> > > > >> >> Thanks for the explanation. Please see the replies inline
> below.
> > > > >> >>
> > > > >> >> I agree. And from my understanding, demoting a Public API is
> > also a
> > > > >> kind of
> > > > >> >> > such change, just like removing one, which can only happen
> with
> > > > major
> > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> APIs
> > > > >> anytime,
> > > > >> >> but
> > > > >> >> > only in the case major version bumps happen before reaching
> the
> > > > >> >> > 2-minor-release migration period. Actually, demoting would
> be a
> > > > weaker
> > > > >> >> > change compared to removing the API immediately upon major
> > > version
> > > > >> bumps,
> > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > migration
> > > > >> >> period.
> > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > > against
> > > > >> >> > conventions, we may introduce a new annotation if necessary,
> > > e.g.,
> > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > >> >>
> > > > >> >> As an end user who only uses Public APIs, if I don't change my
> > code
> > > > at
> > > > >> all,
> > > > >> >> my expectation is the following:
> > > > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > > with
> > > > all
> > > > >> >> the 2.x versions.
> > > > >> >> Actually I think there are some dependency version resolution
> > > > policies
> > > > >> out
> > > > >> >> there which picks the highest minor version when the
> dependencies
> > > > pull
> > > > >> in
> > > > >> >> multiple minor versions of the same jar, which may be broken if
> > we
> > > > >> remove
> > > > >> >> the API in minor releases.
> > > > >> >>
> > > > >> >> I'm not sure about this. Yes, it's completely "legal" that we
> > bump
> > > up
> > > > >> the
> > > > >> >> > major version whenever a breaking change is needed. However,
> > this
> > > > also
> > > > >> >> > weakens the value of the commitment that public APIs will
> stay
> > > > stable
> > > > >> >> > within the major release series, as the series can end
> anytime.
> > > > IMHO,
> > > > >> >> short
> > > > >> >> > major release series are not something "make the end users
> > > happy",
> > > > but
> > > > >> >> > backdoors that allow us as the developers to make frequent
> > > breaking
> > > > >> >> > changes. On the contrary, with the demoting approach, we can
> > > still
> > > > >> have
> > > > >> >> > longer major release series, while only allowing Public APIs
> > > > >> deprecated
> > > > >> >> at
> > > > >> >> > the end of the previous major version to be removed in the
> next
> > > > major
> > > > >> >> > version.
> > > > >> >>
> > > > >> >> I totally agree that frequent major version bumps are not
> ideal,
> > > but
> > > > >> here
> > > > >> >> we are comparing it with a minor version bump which removes a
> > > Public
> > > > >> API.
> > > > >> >> So the context is that we have already decided to remove this
> > > Public
> > > > API
> > > > >> >> while keeping everything else backwards compatible. I think a
> > major
> > > > >> version
> > > > >> >> bump is a commonly understood signal here, compared with a
> minor
> > > > version
> > > > >> >> change. From end users' perspective, for those who are not
> > > impacted,
> > > > in
> > > > >> >> this case upgrading a major version is not necessarily more
> > > involved
> > > > >> than
> > > > >> >> upgrading a minor version - both should be as smooth as a
> > > dependency
> > > > >> >> version change. For those who are impacted, they will lose the
> > > Public
> > > > >> API
> > > > >> >> anyways and a major version bump ensures there is no surprise.
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >>
> > > > >> >> Jiangjie (Becket) Qin
> > > > >> >>
> > > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > tonysong820@gmail.com
> > > > >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Public API is a well defined common concept, and one of its
> > > > >> >> >> convention is that it only changes with a major version
> > change.
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> > I agree. And from my understanding, demoting a Public API is
> > > also a
> > > > >> kind
> > > > >> >> > of such change, just like removing one, which can only happen
> > > with
> > > > >> major
> > > > >> >> > version bumps. I'm not proposing to allow demoting Public
> APIs
> > > > >> anytime,
> > > > >> >> but
> > > > >> >> > only in the case major version bumps happen before reaching
> the
> > > > >> >> > 2-minor-release migration period. Actually, demoting would
> be a
> > > > weaker
> > > > >> >> > change compared to removing the API immediately upon major
> > > version
> > > > >> bumps,
> > > > >> >> > in order to keep the commitment about the 2-minor-release
> > > migration
> > > > >> >> period.
> > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > > against
> > > > >> >> > conventions, we may introduce a new annotation if necessary,
> > > e.g.,
> > > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > > >> >> >
> > > > >> >> > But it should be
> > > > >> >> >> completely OK to bump up the major version if we really want
> > to
> > > > get
> > > > >> rid
> > > > >> >> of
> > > > >> >> >> a public API, right?
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> > I'm not sure about this. Yes, it's completely "legal" that we
> > > bump
> > > > up
> > > > >> the
> > > > >> >> > major version whenever a breaking change is needed. However,
> > this
> > > > also
> > > > >> >> > weakens the value of the commitment that public APIs will
> stay
> > > > stable
> > > > >> >> > within the major release series, as the series can end
> anytime.
> > > > IMHO,
> > > > >> >> short
> > > > >> >> > major release series are not something "make the end users
> > > happy",
> > > > but
> > > > >> >> > backdoors that allow us as the developers to make frequent
> > > breaking
> > > > >> >> > changes. On the contrary, with the demoting approach, we can
> > > still
> > > > >> have
> > > > >> >> > longer major release series, while only allowing Public APIs
> > > > >> deprecated
> > > > >> >> at
> > > > >> >> > the end of the previous major version to be removed in the
> next
> > > > major
> > > > >> >> > version.
> > > > >> >> >
> > > > >> >> > Given our track record I would prefer a regular cycle (1-2
> > years)
> > > > to
> > > > >> >> >> force us to think about this whole topic, and not put it
> again
> > > to
> > > > the
> > > > >> >> >> wayside and giving us (and users) a clear expectation on
> when
> > > > >> breaking
> > > > >> >> >> changes can be made.
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> > +1. I personally think 2-3 years would be a good time for new
> > > major
> > > > >> >> > versions, or longer if there's no breaking changes needed.
> That
> > > > makes
> > > > >> 1-2
> > > > >> >> > year a perfect time to revisit the topic, while leaving us
> more
> > > > time
> > > > >> to
> > > > >> >> > prepare the major release if needed.
> > > > >> >> >
> > > > >> >> > Best,
> > > > >> >> >
> > > > >> >> > Xintong
> > > > >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > > chesnay@apache.org
> > > > >> >
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > >> >> >> > It would be valuable if we can avoid releasing minor
> > versions
> > > > for
> > > > >> >> >> previous
> > > > >> >> >> > major versions.
> > > > >> >> >>
> > > > >> >> >> On paper, /absolutely /agree, but I'm not sure how viable
> that
> > > is
> > > > in
> > > > >> >> >> practice.
> > > > >> >> >>
> > > > >> >> >> On the current 2.0 agenda is potentially dropping support
> for
> > > Java
> > > > >> 8/11,
> > > > >> >> >> which may very well be a problem for our current users.
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > > >> >> >> > Thanks for the feedback and sorry for the confusion about
> > > Public
> > > > >> API
> > > > >> >> >> > deprecation. I just noticed that there was a mistake in
> the
> > > > NOTES
> > > > >> part
> > > > >> >> >> for
> > > > >> >> >> > Public API due to a copy-paste error... I just fixed it.
> > > > >> >> >> I'm very relieved to hear that. Glad to hear that we are on
> > the
> > > > same
> > > > >> >> >> page on that note.
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > > > >> >> >> > But it should be
> > > > >> >> >> > completely OK to bump up the major version if we really
> want
> > > to
> > > > get
> > > > >> >> rid
> > > > >> >> >> of
> > > > >> >> >> > a public API, right?
> > > > >> >> >>
> > > > >> >> >> Technically yes, but look at how long it took to get us to
> > 2.0.
> > > ;)
> > > > >> >> >>
> > > > >> >> >> There's a separate discussion to be had on the cadence of
> > major
> > > > >> releases
> > > > >> >> >> going forward, and there seem to be different opinions on
> > that.
> > > > >> >> >>
> > > > >> >> >> If we take the Kafka example of 2 minor releases between
> major
> > > > ones,
> > > > >> >> >> that for us means that users have to potentially deal with
> > > > breaking
> > > > >> >> >> changes every 6 months, which seems like a lot.
> > > > >> >> >>
> > > > >> >> >> Given our track record I would prefer a regular cycle (1-2
> > > years)
> > > > to
> > > > >> >> >> force us to think about this whole topic, and not put it
> again
> > > to
> > > > the
> > > > >> >> >> wayside and giving us (and users) a clear expectation on
> when
> > > > >> breaking
> > > > >> >> >> changes can be made.
> > > > >> >> >>
> > > > >> >> >> But again, maybe this should be in a separate thread.
> > > > >> >> >>
> > > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > > > >> >> >> > Do you have an example of behavioral change in mind? Not
> > sure
> > > I
> > > > >> fully
> > > > >> >> >> > understand the concern for behavioral change here.
> > > > >> >> >>
> > > > >> >> >> This could be a lot of things. It can be performance in
> > certain
> > > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied
> > upon
> > > > >> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> > > > >> >> >>
> > > > >> >> >> For a concrete example, consider the job submission. A few
> > > > releases
> > > > >> back
> > > > >> >> >> we made changes such that the initialization of the job
> master
> > > > >> happens
> > > > >> >> >> asynchronously.
> > > > >> >> >> This meant the job submission call returns sooner, and the
> job
> > > > state
> > > > >> >> >> enum was extended to cover this state.
> > > > >> >> >> API-wise we consider this a compatible change, but the
> > observed
> > > > >> behavior
> > > > >> >> >> may be different.
> > > > >> >> >>
> > > > >> >> >> Metrics are another example; I believe over time we changed
> > what
> > > > some
> > > > >> >> >> metrics returned a few times.
> > > > >> >> >>
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
>
> The part I don't understand is if we are willing to have a migration
> period, and do a minor version bump to remove an API, what do we lose to do
> a major version bump instead, so we don't break the common versioning
> semantic?
>

I think we are talking about the cases where a major version bump happens
before a deprecated Public API reaches its migration period. So removing
the API with another major version bump means we have two consecutive major
versions in a very short time. And as previously mentioned, having frequent
major version bumps would weaken the value of the commitment "Public API
stay compatible within a major version". I think users also have
expectations about how long a major version should live, which should be at
least a couple of years rather than 1-2 minor releases.

This is another option, but I think it is very likely more expensive than
> simply bumping the major version. And I imagine there will be questions
> like "why this feature is in 1.20, but does not exist in 2.0"?
>

Yes, it's more expensive than simply bumping the major version, but IMHO
it's probably cheaper than carrying the API until the next major version if
we don't bump the major version very soon. And see my reply above on why
not bumping immediately.

Best,

Xintong



On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <be...@gmail.com> wrote:

> Hi Xintong,
>
> Please see the replies below.
>
> I see your point, that users would feel surprised if they find things no
> > longer work when upgrading to another 2.x minor release. However, I'd
> like
> > to point out that PublicEvolving APIs would have the similar problem
> > anyway. So the question is, how do we catch users' attention and make
> sure
> > they are aware that the Public APIs in 1.x may no longer be Public in
> 2.0.
> > There're various ways to do that, e.g., release notes, warnings in logs,
> > etc.
>
> First of all, I am not a fan of removing PublicEvolving APIs in minor
> version changes. Personally speaking, I want them to be also removed in
> major version changes.
>
> Empirically, I rarely see projects with complex API semantic work well.
> Many, if not most, users don't read docs / release notes / warnings in
> logs. I don't think that is their fault. Typically an application developer
> will have to deal with dozens of libraries maintained by various
> communities / groups. It is just too difficult for them to keep track of
> all the specific semantics each project puts there. If you think about it,
> how many of Flink developers read all the release notes of Guava, Apache
> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of them?
> One would probably try bumping the dependency version and see if there is
> an exception and solve them case by case. And if it is a minor version
> bump, one probably does not expect an exception at all. Another example is
> that in Flink we still have so many usage of deprecated methods all over
> the place, and I strongly doubt everyone knows when the source code of
> these methods are deprecated and when they should be removed.
>
> So, the most important principle of API is simple and intuitive. The
> versioning semantic is a simple and universally accepted API stability
> standard. If we ourselves as Flink developers are relying on this for our
> own dependencies. I don't think we can expect more from our users.
>
> Another possible alternative: whenever there's a deprecated Public API that
> > reaches a major version bump before the migration period, and we also
> don't
> > want to carry it for all the next major release series, we may consider
> > releasing more minor releases for the previous major version after the
> > bump. E.g., an Public API is deprecated in 1.19, and then we bump to 2.0,
> > we can release one more 1.20 after 2.0. That should provide users another
> > choice rather than upgrading to 2.0, while satisfying the 2-minor-release
> > migration period.
>
> This is another option, but I think it is very likely more expensive than
> simply bumping the major version. And I imagine there will be questions
> like "why this feature is in 1.20, but does not exist in 2.0"?
>
> I think my major point is, we should not carry APIs deprecated in a
> > previous major version along all the next major version series. I'd like
> to
> > try giving users more commitments, i.e. the migration period, as long as
> it
> > does not prevent us from making breaking changes. If it doesn't work, I'd
> > be in favor of not providing the migration period, but fallback to only
> > guarantee the compatibility within the major version.
>
> The part I don't understand is if we are willing to have a migration
> period, and do a minor version bump to remove an API, what do we lose to do
> a major version bump instead, so we don't break the common versioning
> semantic?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com>
> wrote:
>
> > >
> > > As an end user who only uses Public APIs, if I don't change my code at
> > > all, my expectation is the following:
> > > 1. Upgrading from 1.x to 2.x may have issues.
> > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with
> all
> > > the 2.x versions.
> > > Actually I think there are some dependency version resolution policies
> > out
> > > there which picks the highest minor version when the dependencies pull
> in
> > > multiple minor versions of the same jar, which may be broken if we
> remove
> > > the API in minor releases.
> > >
> >
> > I see your point, that users would feel surprised if they find things no
> > longer work when upgrading to another 2.x minor release. However, I'd
> like
> > to point out that PublicEvolving APIs would have the similar problem
> > anyway. So the question is, how do we catch users' attention and make
> sure
> > they are aware that the Public APIs in 1.x may no longer be Public in
> 2.0.
> > There're various ways to do that, e.g., release notes, warnings in logs,
> > etc.
> >
> > Another possible alternative: whenever there's a deprecated Public API
> that
> > reaches a major version bump before the migration period, and we also
> don't
> > want to carry it for all the next major release series, we may consider
> > releasing more minor releases for the previous major version after the
> > bump. E.g., an Public API is deprecated in 1.19, and then we bump to 2.0,
> > we can release one more 1.20 after 2.0. That should provide users another
> > choice rather than upgrading to 2.0, while satisfying the 2-minor-release
> > migration period.
> >
> > I think my major point is, we should not carry APIs deprecated in a
> > previous major version along all the next major version series. I'd like
> to
> > try giving users more commitments, i.e. the migration period, as long as
> it
> > does not prevent us from making breaking changes. If it doesn't work, I'd
> > be in favor of not providing the migration period, but fallback to only
> > guarantee the compatibility within the major version.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vv...@apache.org>
> wrote:
> >
> > > Hi Becket,
> > >
> > > Thanks for the reply! I’d like to continue the conversation about
> > > compatibility outside of this FLIP thread, but for now, I can accept
> your
> > > decision. It’s certainly an improvement.
> > >
> > > Thanks again,
> > > John
> > >
> > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > Hi John,
> > > >
> > > > Completely agree with all you said.
> > > >
> > > > Can we consider only dropping deprecated APIs in major releases
> across
> > > the
> > > >> board? I understand that Experimental and PublicEvolving APIs are by
> > > >> definition less stable, but it seems like this should be reflected
> in
> > > the
> > > >> required deprecation period alone. I.e. that we must keep them
> around
> > > for
> > > >> at least zero or one minor release, not that we can drop them in a
> > > minor or
> > > >> patch release.
> > > >
> > > > Personally speaking, I would love to do this, for exactly the reason
> > you
> > > > mentioned. However, I did not propose this due to the following
> > reasons:
> > > >
> > > > 1. I am hesitating a little bit about changing the accepted FLIPs too
> > > soon.
> > > > 2. More importantly, to avoid slowing down our development. At this
> > > point,
> > > > Flink still lacks some design / routines to support good API
> > > evolvability /
> > > > extensibility. Just like you said, it takes some time to be good at
> > this.
> > > > In this case, my concern is that only removing Experimental /
> > > > PublicEvolving APIs in major version changes may result in too much
> > > > overhead and dramatically slow down the development of Flink. So, I
> was
> > > > thinking that we can start with the current status. Hopefully after
> we
> > > are
> > > > more comfortable with the maintenance overhead of deprecated APIs, we
> > can
> > > > then have a stronger guarantee for Experimental / PublicEvolving
> APIs.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > >> Hi Becket,
> > > >>
> > > >> Thanks for this FLIP! Having a deprecation process is really
> > important.
> > > I
> > > >> understand some people’s concerns about the additional burden for
> > > project
> > > >> maintainers, but my personal experience with Kafka has been that
> it’s
> > > very
> > > >> liveable and that it’s well worth the benefit to users. In fact,
> users
> > > >> being able to confidently upgrade is also a benefit to maintainers,
> as
> > > we
> > > >> will get fewer questions from people stuck on very old versions.
> > > >>
> > > >> One question:
> > > >> Can we consider only dropping deprecated APIs in major releases
> across
> > > the
> > > >> board? I understand that Experimental and PublicEvolving APIs are by
> > > >> definition less stable, but it seems like this should be reflected
> in
> > > the
> > > >> required deprecation period alone. I.e. that we must keep them
> around
> > > for
> > > >> at least zero or one minor release, not that we can drop them in a
> > > minor or
> > > >> patch release.
> > > >>
> > > >> The advantage of forbidding the removal of any API in minor or patch
> > > >> releases is that users will get a strong guarantee that they can
> bump
> > > the
> > > >> minor or patch version and still be able to compile, or even just
> > > re-link
> > > >> and know that they won’t face “MethodDef” exceptions at run time.
> This
> > > is a
> > > >> binary guarantee: if we allow removing  even Experimental APIs
> outside
> > > of
> > > >> major releases, users can no longer confidently upgrade.
> > > >>
> > > >> Aside from that, I’d share my 2 cents on a couple of points:
> > > >> * I’d use the official Deprecated annotation instead of introducing
> > our
> > > >> own flavor (Retired, etc), since Deprecated is well integrated into
> > > build
> > > >> tools and IDEs.
> > > >> * I wouldn’t worry about a demotion process in this FLIP; it seems
> > > >> orthogonal, and something that should probably be taken case-by-case
> > > >> anyway.
> > > >> * Aside from deprecation and removal, there have been some
> discussions
> > > >> about how to evolve APIs and behavior in compatible ways. This is
> > > somewhat
> > > >> of an art, and if folks haven’t wrestled with it before, it’ll take
> > some
> > > >> time to become good at it. I feel like this topic should also be
> > > orthogonal
> > > >> to this FLIP, but FWIW, my suggestion would be to adopt a simple
> > policy
> > > not
> > > >> to break existing user programs, and leave the “how” up to
> > implementers
> > > and
> > > >> reviewers.
> > > >>
> > > >> Thanks again,
> > > >> John
> > > >>
> > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > >> > Hi All,
> > > >> >
> > > >> > The @Public -> @PublicEvolving proposed by Xintong is a great
> idea.
> > > >> > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving
> > --(2
> > > >> > minor release)--> @Public --> @deprecated --(1 major
> > > >> > release)--> @PublicRetired. It will provide a lot of flexibility
> > > without
> > > >> > breaking any rules we had. @Public APIs are allowed to change
> > between
> > > >> major
> > > >> > releases. Changing annotations is acceptable and provides
> additional
> > > >> > tolerance i.e. user-friendliness, since the APIs themself are not
> > > >> changed.
> > > >> >
> > > >> > I had similar thoughts when I was facing those issues. I want to
> > move
> > > one
> > > >> > step further and suggest introducing one more annotation @Retired.
> > > >> >
> > > >> > Not like the @PublicRetired which is a compromise of downgrading
> > > @Public
> > > >> to
> > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java standard
> > > >> > @deprecated should be used in the early stage of the deprecation
> > > process
> > > >> > and doesn't really meet our requirement. Since Java does not allow
> > us
> > > to
> > > >> > extend annotation, I think it would be feasible to have the new
> > > @Retired
> > > >> to
> > > >> > help us monitor and manage the deprecation process, house
> cleaning,
> > > etc.
> > > >> >
> > > >> > Some ideas could be(open for discussion):
> > > >> >
> > > >> > @Retired:
> > > >> >
> > > >> > 1. There must be a replacement with functionality compatibility
> > before
> > > >> APIs
> > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the
> ML
> > > are
> > > >> > mandatory (a FLIP is recommended).
> > > >> > 2. APIs marked as @Retired will be removed after 1 minor release
> > > sharply
> > > >> > (using ArchUnit to force it, needs further check whether it is
> > > possible).
> > > >> > Devs who marked them as @Retired are responsible to remove them.
> > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> > > >> > recommended. @Experimental -> @Retired and @Internal -> @Retired
> > could
> > > >> also
> > > >> > be used if it can increase user-friendliness or dev-friendliness,
> > but
> > > not
> > > >> > mandatory.
> > > >> > 4. Some variables will be defined in @Retired to support the
> > > deprecation
> > > >> > process management. Further extension is possible, since the
> > > annotation
> > > >> is
> > > >> > built by us.
> > > >> >
> > > >> >
> > > >> > Best regards,
> > > >> > Jing
> > > >> >
> > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <becket.qin@gmail.com
> >
> > > >> wrote:
> > > >> >
> > > >> >> Hi Xintong,
> > > >> >>
> > > >> >> Thanks for the explanation. Please see the replies inline below.
> > > >> >>
> > > >> >> I agree. And from my understanding, demoting a Public API is
> also a
> > > >> kind of
> > > >> >> > such change, just like removing one, which can only happen with
> > > major
> > > >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> > > >> anytime,
> > > >> >> but
> > > >> >> > only in the case major version bumps happen before reaching the
> > > >> >> > 2-minor-release migration period. Actually, demoting would be a
> > > weaker
> > > >> >> > change compared to removing the API immediately upon major
> > version
> > > >> bumps,
> > > >> >> > in order to keep the commitment about the 2-minor-release
> > migration
> > > >> >> period.
> > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > against
> > > >> >> > conventions, we may introduce a new annotation if necessary,
> > e.g.,
> > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > >> >>
> > > >> >> As an end user who only uses Public APIs, if I don't change my
> code
> > > at
> > > >> all,
> > > >> >> my expectation is the following:
> > > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> > with
> > > all
> > > >> >> the 2.x versions.
> > > >> >> Actually I think there are some dependency version resolution
> > > policies
> > > >> out
> > > >> >> there which picks the highest minor version when the dependencies
> > > pull
> > > >> in
> > > >> >> multiple minor versions of the same jar, which may be broken if
> we
> > > >> remove
> > > >> >> the API in minor releases.
> > > >> >>
> > > >> >> I'm not sure about this. Yes, it's completely "legal" that we
> bump
> > up
> > > >> the
> > > >> >> > major version whenever a breaking change is needed. However,
> this
> > > also
> > > >> >> > weakens the value of the commitment that public APIs will stay
> > > stable
> > > >> >> > within the major release series, as the series can end anytime.
> > > IMHO,
> > > >> >> short
> > > >> >> > major release series are not something "make the end users
> > happy",
> > > but
> > > >> >> > backdoors that allow us as the developers to make frequent
> > breaking
> > > >> >> > changes. On the contrary, with the demoting approach, we can
> > still
> > > >> have
> > > >> >> > longer major release series, while only allowing Public APIs
> > > >> deprecated
> > > >> >> at
> > > >> >> > the end of the previous major version to be removed in the next
> > > major
> > > >> >> > version.
> > > >> >>
> > > >> >> I totally agree that frequent major version bumps are not ideal,
> > but
> > > >> here
> > > >> >> we are comparing it with a minor version bump which removes a
> > Public
> > > >> API.
> > > >> >> So the context is that we have already decided to remove this
> > Public
> > > API
> > > >> >> while keeping everything else backwards compatible. I think a
> major
> > > >> version
> > > >> >> bump is a commonly understood signal here, compared with a minor
> > > version
> > > >> >> change. From end users' perspective, for those who are not
> > impacted,
> > > in
> > > >> >> this case upgrading a major version is not necessarily more
> > involved
> > > >> than
> > > >> >> upgrading a minor version - both should be as smooth as a
> > dependency
> > > >> >> version change. For those who are impacted, they will lose the
> > Public
> > > >> API
> > > >> >> anyways and a major version bump ensures there is no surprise.
> > > >> >>
> > > >> >> Thanks,
> > > >> >>
> > > >> >> Jiangjie (Becket) Qin
> > > >> >>
> > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > tonysong820@gmail.com
> > > >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Public API is a well defined common concept, and one of its
> > > >> >> >> convention is that it only changes with a major version
> change.
> > > >> >> >>
> > > >> >> >
> > > >> >> > I agree. And from my understanding, demoting a Public API is
> > also a
> > > >> kind
> > > >> >> > of such change, just like removing one, which can only happen
> > with
> > > >> major
> > > >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> > > >> anytime,
> > > >> >> but
> > > >> >> > only in the case major version bumps happen before reaching the
> > > >> >> > 2-minor-release migration period. Actually, demoting would be a
> > > weaker
> > > >> >> > change compared to removing the API immediately upon major
> > version
> > > >> bumps,
> > > >> >> > in order to keep the commitment about the 2-minor-release
> > migration
> > > >> >> period.
> > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > > against
> > > >> >> > conventions, we may introduce a new annotation if necessary,
> > e.g.,
> > > >> >> > `@PublicRetiring`, to avoid confusions.
> > > >> >> >
> > > >> >> > But it should be
> > > >> >> >> completely OK to bump up the major version if we really want
> to
> > > get
> > > >> rid
> > > >> >> of
> > > >> >> >> a public API, right?
> > > >> >> >>
> > > >> >> >
> > > >> >> > I'm not sure about this. Yes, it's completely "legal" that we
> > bump
> > > up
> > > >> the
> > > >> >> > major version whenever a breaking change is needed. However,
> this
> > > also
> > > >> >> > weakens the value of the commitment that public APIs will stay
> > > stable
> > > >> >> > within the major release series, as the series can end anytime.
> > > IMHO,
> > > >> >> short
> > > >> >> > major release series are not something "make the end users
> > happy",
> > > but
> > > >> >> > backdoors that allow us as the developers to make frequent
> > breaking
> > > >> >> > changes. On the contrary, with the demoting approach, we can
> > still
> > > >> have
> > > >> >> > longer major release series, while only allowing Public APIs
> > > >> deprecated
> > > >> >> at
> > > >> >> > the end of the previous major version to be removed in the next
> > > major
> > > >> >> > version.
> > > >> >> >
> > > >> >> > Given our track record I would prefer a regular cycle (1-2
> years)
> > > to
> > > >> >> >> force us to think about this whole topic, and not put it again
> > to
> > > the
> > > >> >> >> wayside and giving us (and users) a clear expectation on when
> > > >> breaking
> > > >> >> >> changes can be made.
> > > >> >> >>
> > > >> >> >
> > > >> >> > +1. I personally think 2-3 years would be a good time for new
> > major
> > > >> >> > versions, or longer if there's no breaking changes needed. That
> > > makes
> > > >> 1-2
> > > >> >> > year a perfect time to revisit the topic, while leaving us more
> > > time
> > > >> to
> > > >> >> > prepare the major release if needed.
> > > >> >> >
> > > >> >> > Best,
> > > >> >> >
> > > >> >> > Xintong
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > > chesnay@apache.org
> > > >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > >> >> >> > It would be valuable if we can avoid releasing minor
> versions
> > > for
> > > >> >> >> previous
> > > >> >> >> > major versions.
> > > >> >> >>
> > > >> >> >> On paper, /absolutely /agree, but I'm not sure how viable that
> > is
> > > in
> > > >> >> >> practice.
> > > >> >> >>
> > > >> >> >> On the current 2.0 agenda is potentially dropping support for
> > Java
> > > >> 8/11,
> > > >> >> >> which may very well be a problem for our current users.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > > >> >> >> > Thanks for the feedback and sorry for the confusion about
> > Public
> > > >> API
> > > >> >> >> > deprecation. I just noticed that there was a mistake in the
> > > NOTES
> > > >> part
> > > >> >> >> for
> > > >> >> >> > Public API due to a copy-paste error... I just fixed it.
> > > >> >> >> I'm very relieved to hear that. Glad to hear that we are on
> the
> > > same
> > > >> >> >> page on that note.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > > >> >> >> > But it should be
> > > >> >> >> > completely OK to bump up the major version if we really want
> > to
> > > get
> > > >> >> rid
> > > >> >> >> of
> > > >> >> >> > a public API, right?
> > > >> >> >>
> > > >> >> >> Technically yes, but look at how long it took to get us to
> 2.0.
> > ;)
> > > >> >> >>
> > > >> >> >> There's a separate discussion to be had on the cadence of
> major
> > > >> releases
> > > >> >> >> going forward, and there seem to be different opinions on
> that.
> > > >> >> >>
> > > >> >> >> If we take the Kafka example of 2 minor releases between major
> > > ones,
> > > >> >> >> that for us means that users have to potentially deal with
> > > breaking
> > > >> >> >> changes every 6 months, which seems like a lot.
> > > >> >> >>
> > > >> >> >> Given our track record I would prefer a regular cycle (1-2
> > years)
> > > to
> > > >> >> >> force us to think about this whole topic, and not put it again
> > to
> > > the
> > > >> >> >> wayside and giving us (and users) a clear expectation on when
> > > >> breaking
> > > >> >> >> changes can be made.
> > > >> >> >>
> > > >> >> >> But again, maybe this should be in a separate thread.
> > > >> >> >>
> > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > > >> >> >> > Do you have an example of behavioral change in mind? Not
> sure
> > I
> > > >> fully
> > > >> >> >> > understand the concern for behavioral change here.
> > > >> >> >>
> > > >> >> >> This could be a lot of things. It can be performance in
> certain
> > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied
> upon
> > > >> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> > > >> >> >>
> > > >> >> >> For a concrete example, consider the job submission. A few
> > > releases
> > > >> back
> > > >> >> >> we made changes such that the initialization of the job master
> > > >> happens
> > > >> >> >> asynchronously.
> > > >> >> >> This meant the job submission call returns sooner, and the job
> > > state
> > > >> >> >> enum was extended to cover this state.
> > > >> >> >> API-wise we consider this a compatible change, but the
> observed
> > > >> behavior
> > > >> >> >> may be different.
> > > >> >> >>
> > > >> >> >> Metrics are another example; I believe over time we changed
> what
> > > some
> > > >> >> >> metrics returned a few times.
> > > >> >> >>
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

Please see the replies below.

I see your point, that users would feel surprised if they find things no
> longer work when upgrading to another 2.x minor release. However, I'd like
> to point out that PublicEvolving APIs would have the similar problem
> anyway. So the question is, how do we catch users' attention and make sure
> they are aware that the Public APIs in 1.x may no longer be Public in 2.0.
> There're various ways to do that, e.g., release notes, warnings in logs,
> etc.

First of all, I am not a fan of removing PublicEvolving APIs in minor
version changes. Personally speaking, I want them to be also removed in
major version changes.

Empirically, I rarely see projects with complex API semantic work well.
Many, if not most, users don't read docs / release notes / warnings in
logs. I don't think that is their fault. Typically an application developer
will have to deal with dozens of libraries maintained by various
communities / groups. It is just too difficult for them to keep track of
all the specific semantics each project puts there. If you think about it,
how many of Flink developers read all the release notes of Guava, Apache
Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of them?
One would probably try bumping the dependency version and see if there is
an exception and solve them case by case. And if it is a minor version
bump, one probably does not expect an exception at all. Another example is
that in Flink we still have so many usage of deprecated methods all over
the place, and I strongly doubt everyone knows when the source code of
these methods are deprecated and when they should be removed.

So, the most important principle of API is simple and intuitive. The
versioning semantic is a simple and universally accepted API stability
standard. If we ourselves as Flink developers are relying on this for our
own dependencies. I don't think we can expect more from our users.

Another possible alternative: whenever there's a deprecated Public API that
> reaches a major version bump before the migration period, and we also don't
> want to carry it for all the next major release series, we may consider
> releasing more minor releases for the previous major version after the
> bump. E.g., an Public API is deprecated in 1.19, and then we bump to 2.0,
> we can release one more 1.20 after 2.0. That should provide users another
> choice rather than upgrading to 2.0, while satisfying the 2-minor-release
> migration period.

This is another option, but I think it is very likely more expensive than
simply bumping the major version. And I imagine there will be questions
like "why this feature is in 1.20, but does not exist in 2.0"?

I think my major point is, we should not carry APIs deprecated in a
> previous major version along all the next major version series. I'd like to
> try giving users more commitments, i.e. the migration period, as long as it
> does not prevent us from making breaking changes. If it doesn't work, I'd
> be in favor of not providing the migration period, but fallback to only
> guarantee the compatibility within the major version.

The part I don't understand is if we are willing to have a migration
period, and do a minor version bump to remove an API, what do we lose to do
a major version bump instead, so we don't break the common versioning
semantic?

Thanks,

Jiangjie (Becket) Qin


On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <to...@gmail.com> wrote:

> >
> > As an end user who only uses Public APIs, if I don't change my code at
> > all, my expectation is the following:
> > 1. Upgrading from 1.x to 2.x may have issues.
> > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
> > the 2.x versions.
> > Actually I think there are some dependency version resolution policies
> out
> > there which picks the highest minor version when the dependencies pull in
> > multiple minor versions of the same jar, which may be broken if we remove
> > the API in minor releases.
> >
>
> I see your point, that users would feel surprised if they find things no
> longer work when upgrading to another 2.x minor release. However, I'd like
> to point out that PublicEvolving APIs would have the similar problem
> anyway. So the question is, how do we catch users' attention and make sure
> they are aware that the Public APIs in 1.x may no longer be Public in 2.0.
> There're various ways to do that, e.g., release notes, warnings in logs,
> etc.
>
> Another possible alternative: whenever there's a deprecated Public API that
> reaches a major version bump before the migration period, and we also don't
> want to carry it for all the next major release series, we may consider
> releasing more minor releases for the previous major version after the
> bump. E.g., an Public API is deprecated in 1.19, and then we bump to 2.0,
> we can release one more 1.20 after 2.0. That should provide users another
> choice rather than upgrading to 2.0, while satisfying the 2-minor-release
> migration period.
>
> I think my major point is, we should not carry APIs deprecated in a
> previous major version along all the next major version series. I'd like to
> try giving users more commitments, i.e. the migration period, as long as it
> does not prevent us from making breaking changes. If it doesn't work, I'd
> be in favor of not providing the migration period, but fallback to only
> guarantee the compatibility within the major version.
>
> Best,
>
> Xintong
>
>
>
> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vv...@apache.org> wrote:
>
> > Hi Becket,
> >
> > Thanks for the reply! I’d like to continue the conversation about
> > compatibility outside of this FLIP thread, but for now, I can accept your
> > decision. It’s certainly an improvement.
> >
> > Thanks again,
> > John
> >
> > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > Hi John,
> > >
> > > Completely agree with all you said.
> > >
> > > Can we consider only dropping deprecated APIs in major releases across
> > the
> > >> board? I understand that Experimental and PublicEvolving APIs are by
> > >> definition less stable, but it seems like this should be reflected in
> > the
> > >> required deprecation period alone. I.e. that we must keep them around
> > for
> > >> at least zero or one minor release, not that we can drop them in a
> > minor or
> > >> patch release.
> > >
> > > Personally speaking, I would love to do this, for exactly the reason
> you
> > > mentioned. However, I did not propose this due to the following
> reasons:
> > >
> > > 1. I am hesitating a little bit about changing the accepted FLIPs too
> > soon.
> > > 2. More importantly, to avoid slowing down our development. At this
> > point,
> > > Flink still lacks some design / routines to support good API
> > evolvability /
> > > extensibility. Just like you said, it takes some time to be good at
> this.
> > > In this case, my concern is that only removing Experimental /
> > > PublicEvolving APIs in major version changes may result in too much
> > > overhead and dramatically slow down the development of Flink. So, I was
> > > thinking that we can start with the current status. Hopefully after we
> > are
> > > more comfortable with the maintenance overhead of deprecated APIs, we
> can
> > > then have a stronger guarantee for Experimental / PublicEvolving APIs.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > >> Hi Becket,
> > >>
> > >> Thanks for this FLIP! Having a deprecation process is really
> important.
> > I
> > >> understand some people’s concerns about the additional burden for
> > project
> > >> maintainers, but my personal experience with Kafka has been that it’s
> > very
> > >> liveable and that it’s well worth the benefit to users. In fact, users
> > >> being able to confidently upgrade is also a benefit to maintainers, as
> > we
> > >> will get fewer questions from people stuck on very old versions.
> > >>
> > >> One question:
> > >> Can we consider only dropping deprecated APIs in major releases across
> > the
> > >> board? I understand that Experimental and PublicEvolving APIs are by
> > >> definition less stable, but it seems like this should be reflected in
> > the
> > >> required deprecation period alone. I.e. that we must keep them around
> > for
> > >> at least zero or one minor release, not that we can drop them in a
> > minor or
> > >> patch release.
> > >>
> > >> The advantage of forbidding the removal of any API in minor or patch
> > >> releases is that users will get a strong guarantee that they can bump
> > the
> > >> minor or patch version and still be able to compile, or even just
> > re-link
> > >> and know that they won’t face “MethodDef” exceptions at run time. This
> > is a
> > >> binary guarantee: if we allow removing  even Experimental APIs outside
> > of
> > >> major releases, users can no longer confidently upgrade.
> > >>
> > >> Aside from that, I’d share my 2 cents on a couple of points:
> > >> * I’d use the official Deprecated annotation instead of introducing
> our
> > >> own flavor (Retired, etc), since Deprecated is well integrated into
> > build
> > >> tools and IDEs.
> > >> * I wouldn’t worry about a demotion process in this FLIP; it seems
> > >> orthogonal, and something that should probably be taken case-by-case
> > >> anyway.
> > >> * Aside from deprecation and removal, there have been some discussions
> > >> about how to evolve APIs and behavior in compatible ways. This is
> > somewhat
> > >> of an art, and if folks haven’t wrestled with it before, it’ll take
> some
> > >> time to become good at it. I feel like this topic should also be
> > orthogonal
> > >> to this FLIP, but FWIW, my suggestion would be to adopt a simple
> policy
> > not
> > >> to break existing user programs, and leave the “how” up to
> implementers
> > and
> > >> reviewers.
> > >>
> > >> Thanks again,
> > >> John
> > >>
> > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > >> > Hi All,
> > >> >
> > >> > The @Public -> @PublicEvolving proposed by Xintong is a great idea.
> > >> > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving
> --(2
> > >> > minor release)--> @Public --> @deprecated --(1 major
> > >> > release)--> @PublicRetired. It will provide a lot of flexibility
> > without
> > >> > breaking any rules we had. @Public APIs are allowed to change
> between
> > >> major
> > >> > releases. Changing annotations is acceptable and provides additional
> > >> > tolerance i.e. user-friendliness, since the APIs themself are not
> > >> changed.
> > >> >
> > >> > I had similar thoughts when I was facing those issues. I want to
> move
> > one
> > >> > step further and suggest introducing one more annotation @Retired.
> > >> >
> > >> > Not like the @PublicRetired which is a compromise of downgrading
> > @Public
> > >> to
> > >> > @PublicEvolving. As I mentioned earlier in my reply, Java standard
> > >> > @deprecated should be used in the early stage of the deprecation
> > process
> > >> > and doesn't really meet our requirement. Since Java does not allow
> us
> > to
> > >> > extend annotation, I think it would be feasible to have the new
> > @Retired
> > >> to
> > >> > help us monitor and manage the deprecation process, house cleaning,
> > etc.
> > >> >
> > >> > Some ideas could be(open for discussion):
> > >> >
> > >> > @Retired:
> > >> >
> > >> > 1. There must be a replacement with functionality compatibility
> before
> > >> APIs
> > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML
> > are
> > >> > mandatory (a FLIP is recommended).
> > >> > 2. APIs marked as @Retired will be removed after 1 minor release
> > sharply
> > >> > (using ArchUnit to force it, needs further check whether it is
> > possible).
> > >> > Devs who marked them as @Retired are responsible to remove them.
> > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> > >> > recommended. @Experimental -> @Retired and @Internal -> @Retired
> could
> > >> also
> > >> > be used if it can increase user-friendliness or dev-friendliness,
> but
> > not
> > >> > mandatory.
> > >> > 4. Some variables will be defined in @Retired to support the
> > deprecation
> > >> > process management. Further extension is possible, since the
> > annotation
> > >> is
> > >> > built by us.
> > >> >
> > >> >
> > >> > Best regards,
> > >> > Jing
> > >> >
> > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> Hi Xintong,
> > >> >>
> > >> >> Thanks for the explanation. Please see the replies inline below.
> > >> >>
> > >> >> I agree. And from my understanding, demoting a Public API is also a
> > >> kind of
> > >> >> > such change, just like removing one, which can only happen with
> > major
> > >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> > >> anytime,
> > >> >> but
> > >> >> > only in the case major version bumps happen before reaching the
> > >> >> > 2-minor-release migration period. Actually, demoting would be a
> > weaker
> > >> >> > change compared to removing the API immediately upon major
> version
> > >> bumps,
> > >> >> > in order to keep the commitment about the 2-minor-release
> migration
> > >> >> period.
> > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > against
> > >> >> > conventions, we may introduce a new annotation if necessary,
> e.g.,
> > >> >> > `@PublicRetiring`, to avoid confusions.
> > >> >>
> > >> >> As an end user who only uses Public APIs, if I don't change my code
> > at
> > >> all,
> > >> >> my expectation is the following:
> > >> >> 1. Upgrading from 1.x to 2.x may have issues.
> > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine
> with
> > all
> > >> >> the 2.x versions.
> > >> >> Actually I think there are some dependency version resolution
> > policies
> > >> out
> > >> >> there which picks the highest minor version when the dependencies
> > pull
> > >> in
> > >> >> multiple minor versions of the same jar, which may be broken if we
> > >> remove
> > >> >> the API in minor releases.
> > >> >>
> > >> >> I'm not sure about this. Yes, it's completely "legal" that we bump
> up
> > >> the
> > >> >> > major version whenever a breaking change is needed. However, this
> > also
> > >> >> > weakens the value of the commitment that public APIs will stay
> > stable
> > >> >> > within the major release series, as the series can end anytime.
> > IMHO,
> > >> >> short
> > >> >> > major release series are not something "make the end users
> happy",
> > but
> > >> >> > backdoors that allow us as the developers to make frequent
> breaking
> > >> >> > changes. On the contrary, with the demoting approach, we can
> still
> > >> have
> > >> >> > longer major release series, while only allowing Public APIs
> > >> deprecated
> > >> >> at
> > >> >> > the end of the previous major version to be removed in the next
> > major
> > >> >> > version.
> > >> >>
> > >> >> I totally agree that frequent major version bumps are not ideal,
> but
> > >> here
> > >> >> we are comparing it with a minor version bump which removes a
> Public
> > >> API.
> > >> >> So the context is that we have already decided to remove this
> Public
> > API
> > >> >> while keeping everything else backwards compatible. I think a major
> > >> version
> > >> >> bump is a commonly understood signal here, compared with a minor
> > version
> > >> >> change. From end users' perspective, for those who are not
> impacted,
> > in
> > >> >> this case upgrading a major version is not necessarily more
> involved
> > >> than
> > >> >> upgrading a minor version - both should be as smooth as a
> dependency
> > >> >> version change. For those who are impacted, they will lose the
> Public
> > >> API
> > >> >> anyways and a major version bump ensures there is no surprise.
> > >> >>
> > >> >> Thanks,
> > >> >>
> > >> >> Jiangjie (Becket) Qin
> > >> >>
> > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> tonysong820@gmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > Public API is a well defined common concept, and one of its
> > >> >> >> convention is that it only changes with a major version change.
> > >> >> >>
> > >> >> >
> > >> >> > I agree. And from my understanding, demoting a Public API is
> also a
> > >> kind
> > >> >> > of such change, just like removing one, which can only happen
> with
> > >> major
> > >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> > >> anytime,
> > >> >> but
> > >> >> > only in the case major version bumps happen before reaching the
> > >> >> > 2-minor-release migration period. Actually, demoting would be a
> > weaker
> > >> >> > change compared to removing the API immediately upon major
> version
> > >> bumps,
> > >> >> > in order to keep the commitment about the 2-minor-release
> migration
> > >> >> period.
> > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> > against
> > >> >> > conventions, we may introduce a new annotation if necessary,
> e.g.,
> > >> >> > `@PublicRetiring`, to avoid confusions.
> > >> >> >
> > >> >> > But it should be
> > >> >> >> completely OK to bump up the major version if we really want to
> > get
> > >> rid
> > >> >> of
> > >> >> >> a public API, right?
> > >> >> >>
> > >> >> >
> > >> >> > I'm not sure about this. Yes, it's completely "legal" that we
> bump
> > up
> > >> the
> > >> >> > major version whenever a breaking change is needed. However, this
> > also
> > >> >> > weakens the value of the commitment that public APIs will stay
> > stable
> > >> >> > within the major release series, as the series can end anytime.
> > IMHO,
> > >> >> short
> > >> >> > major release series are not something "make the end users
> happy",
> > but
> > >> >> > backdoors that allow us as the developers to make frequent
> breaking
> > >> >> > changes. On the contrary, with the demoting approach, we can
> still
> > >> have
> > >> >> > longer major release series, while only allowing Public APIs
> > >> deprecated
> > >> >> at
> > >> >> > the end of the previous major version to be removed in the next
> > major
> > >> >> > version.
> > >> >> >
> > >> >> > Given our track record I would prefer a regular cycle (1-2 years)
> > to
> > >> >> >> force us to think about this whole topic, and not put it again
> to
> > the
> > >> >> >> wayside and giving us (and users) a clear expectation on when
> > >> breaking
> > >> >> >> changes can be made.
> > >> >> >>
> > >> >> >
> > >> >> > +1. I personally think 2-3 years would be a good time for new
> major
> > >> >> > versions, or longer if there's no breaking changes needed. That
> > makes
> > >> 1-2
> > >> >> > year a perfect time to revisit the topic, while leaving us more
> > time
> > >> to
> > >> >> > prepare the major release if needed.
> > >> >> >
> > >> >> > Best,
> > >> >> >
> > >> >> > Xintong
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> > chesnay@apache.org
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > >> >> >> > It would be valuable if we can avoid releasing minor versions
> > for
> > >> >> >> previous
> > >> >> >> > major versions.
> > >> >> >>
> > >> >> >> On paper, /absolutely /agree, but I'm not sure how viable that
> is
> > in
> > >> >> >> practice.
> > >> >> >>
> > >> >> >> On the current 2.0 agenda is potentially dropping support for
> Java
> > >> 8/11,
> > >> >> >> which may very well be a problem for our current users.
> > >> >> >>
> > >> >> >>
> > >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> > >> >> >> > Thanks for the feedback and sorry for the confusion about
> Public
> > >> API
> > >> >> >> > deprecation. I just noticed that there was a mistake in the
> > NOTES
> > >> part
> > >> >> >> for
> > >> >> >> > Public API due to a copy-paste error... I just fixed it.
> > >> >> >> I'm very relieved to hear that. Glad to hear that we are on the
> > same
> > >> >> >> page on that note.
> > >> >> >>
> > >> >> >>
> > >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> > >> >> >> > But it should be
> > >> >> >> > completely OK to bump up the major version if we really want
> to
> > get
> > >> >> rid
> > >> >> >> of
> > >> >> >> > a public API, right?
> > >> >> >>
> > >> >> >> Technically yes, but look at how long it took to get us to 2.0.
> ;)
> > >> >> >>
> > >> >> >> There's a separate discussion to be had on the cadence of major
> > >> releases
> > >> >> >> going forward, and there seem to be different opinions on that.
> > >> >> >>
> > >> >> >> If we take the Kafka example of 2 minor releases between major
> > ones,
> > >> >> >> that for us means that users have to potentially deal with
> > breaking
> > >> >> >> changes every 6 months, which seems like a lot.
> > >> >> >>
> > >> >> >> Given our track record I would prefer a regular cycle (1-2
> years)
> > to
> > >> >> >> force us to think about this whole topic, and not put it again
> to
> > the
> > >> >> >> wayside and giving us (and users) a clear expectation on when
> > >> breaking
> > >> >> >> changes can be made.
> > >> >> >>
> > >> >> >> But again, maybe this should be in a separate thread.
> > >> >> >>
> > >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> > >> >> >> > Do you have an example of behavioral change in mind? Not sure
> I
> > >> fully
> > >> >> >> > understand the concern for behavioral change here.
> > >> >> >>
> > >> >> >> This could be a lot of things. It can be performance in certain
> > >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
> > >> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> > >> >> >>
> > >> >> >> For a concrete example, consider the job submission. A few
> > releases
> > >> back
> > >> >> >> we made changes such that the initialization of the job master
> > >> happens
> > >> >> >> asynchronously.
> > >> >> >> This meant the job submission call returns sooner, and the job
> > state
> > >> >> >> enum was extended to cover this state.
> > >> >> >> API-wise we consider this a compatible change, but the observed
> > >> behavior
> > >> >> >> may be different.
> > >> >> >>
> > >> >> >> Metrics are another example; I believe over time we changed what
> > some
> > >> >> >> metrics returned a few times.
> > >> >> >>
> > >> >> >
> > >> >>
> > >>
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
>
> As an end user who only uses Public APIs, if I don't change my code at
> all, my expectation is the following:
> 1. Upgrading from 1.x to 2.x may have issues.
> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
> the 2.x versions.
> Actually I think there are some dependency version resolution policies out
> there which picks the highest minor version when the dependencies pull in
> multiple minor versions of the same jar, which may be broken if we remove
> the API in minor releases.
>

I see your point, that users would feel surprised if they find things no
longer work when upgrading to another 2.x minor release. However, I'd like
to point out that PublicEvolving APIs would have the similar problem
anyway. So the question is, how do we catch users' attention and make sure
they are aware that the Public APIs in 1.x may no longer be Public in 2.0.
There're various ways to do that, e.g., release notes, warnings in logs,
etc.

Another possible alternative: whenever there's a deprecated Public API that
reaches a major version bump before the migration period, and we also don't
want to carry it for all the next major release series, we may consider
releasing more minor releases for the previous major version after the
bump. E.g., an Public API is deprecated in 1.19, and then we bump to 2.0,
we can release one more 1.20 after 2.0. That should provide users another
choice rather than upgrading to 2.0, while satisfying the 2-minor-release
migration period.

I think my major point is, we should not carry APIs deprecated in a
previous major version along all the next major version series. I'd like to
try giving users more commitments, i.e. the migration period, as long as it
does not prevent us from making breaking changes. If it doesn't work, I'd
be in favor of not providing the migration period, but fallback to only
guarantee the compatibility within the major version.

Best,

Xintong



On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vv...@apache.org> wrote:

> Hi Becket,
>
> Thanks for the reply! I’d like to continue the conversation about
> compatibility outside of this FLIP thread, but for now, I can accept your
> decision. It’s certainly an improvement.
>
> Thanks again,
> John
>
> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > Hi John,
> >
> > Completely agree with all you said.
> >
> > Can we consider only dropping deprecated APIs in major releases across
> the
> >> board? I understand that Experimental and PublicEvolving APIs are by
> >> definition less stable, but it seems like this should be reflected in
> the
> >> required deprecation period alone. I.e. that we must keep them around
> for
> >> at least zero or one minor release, not that we can drop them in a
> minor or
> >> patch release.
> >
> > Personally speaking, I would love to do this, for exactly the reason you
> > mentioned. However, I did not propose this due to the following reasons:
> >
> > 1. I am hesitating a little bit about changing the accepted FLIPs too
> soon.
> > 2. More importantly, to avoid slowing down our development. At this
> point,
> > Flink still lacks some design / routines to support good API
> evolvability /
> > extensibility. Just like you said, it takes some time to be good at this.
> > In this case, my concern is that only removing Experimental /
> > PublicEvolving APIs in major version changes may result in too much
> > overhead and dramatically slow down the development of Flink. So, I was
> > thinking that we can start with the current status. Hopefully after we
> are
> > more comfortable with the maintenance overhead of deprecated APIs, we can
> > then have a stronger guarantee for Experimental / PublicEvolving APIs.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org>
> wrote:
> >
> >> Hi Becket,
> >>
> >> Thanks for this FLIP! Having a deprecation process is really important.
> I
> >> understand some people’s concerns about the additional burden for
> project
> >> maintainers, but my personal experience with Kafka has been that it’s
> very
> >> liveable and that it’s well worth the benefit to users. In fact, users
> >> being able to confidently upgrade is also a benefit to maintainers, as
> we
> >> will get fewer questions from people stuck on very old versions.
> >>
> >> One question:
> >> Can we consider only dropping deprecated APIs in major releases across
> the
> >> board? I understand that Experimental and PublicEvolving APIs are by
> >> definition less stable, but it seems like this should be reflected in
> the
> >> required deprecation period alone. I.e. that we must keep them around
> for
> >> at least zero or one minor release, not that we can drop them in a
> minor or
> >> patch release.
> >>
> >> The advantage of forbidding the removal of any API in minor or patch
> >> releases is that users will get a strong guarantee that they can bump
> the
> >> minor or patch version and still be able to compile, or even just
> re-link
> >> and know that they won’t face “MethodDef” exceptions at run time. This
> is a
> >> binary guarantee: if we allow removing  even Experimental APIs outside
> of
> >> major releases, users can no longer confidently upgrade.
> >>
> >> Aside from that, I’d share my 2 cents on a couple of points:
> >> * I’d use the official Deprecated annotation instead of introducing our
> >> own flavor (Retired, etc), since Deprecated is well integrated into
> build
> >> tools and IDEs.
> >> * I wouldn’t worry about a demotion process in this FLIP; it seems
> >> orthogonal, and something that should probably be taken case-by-case
> >> anyway.
> >> * Aside from deprecation and removal, there have been some discussions
> >> about how to evolve APIs and behavior in compatible ways. This is
> somewhat
> >> of an art, and if folks haven’t wrestled with it before, it’ll take some
> >> time to become good at it. I feel like this topic should also be
> orthogonal
> >> to this FLIP, but FWIW, my suggestion would be to adopt a simple policy
> not
> >> to break existing user programs, and leave the “how” up to implementers
> and
> >> reviewers.
> >>
> >> Thanks again,
> >> John
> >>
> >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> >> > Hi All,
> >> >
> >> > The @Public -> @PublicEvolving proposed by Xintong is a great idea.
> >> > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
> >> > minor release)--> @Public --> @deprecated --(1 major
> >> > release)--> @PublicRetired. It will provide a lot of flexibility
> without
> >> > breaking any rules we had. @Public APIs are allowed to change between
> >> major
> >> > releases. Changing annotations is acceptable and provides additional
> >> > tolerance i.e. user-friendliness, since the APIs themself are not
> >> changed.
> >> >
> >> > I had similar thoughts when I was facing those issues. I want to move
> one
> >> > step further and suggest introducing one more annotation @Retired.
> >> >
> >> > Not like the @PublicRetired which is a compromise of downgrading
> @Public
> >> to
> >> > @PublicEvolving. As I mentioned earlier in my reply, Java standard
> >> > @deprecated should be used in the early stage of the deprecation
> process
> >> > and doesn't really meet our requirement. Since Java does not allow us
> to
> >> > extend annotation, I think it would be feasible to have the new
> @Retired
> >> to
> >> > help us monitor and manage the deprecation process, house cleaning,
> etc.
> >> >
> >> > Some ideas could be(open for discussion):
> >> >
> >> > @Retired:
> >> >
> >> > 1. There must be a replacement with functionality compatibility before
> >> APIs
> >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML
> are
> >> > mandatory (a FLIP is recommended).
> >> > 2. APIs marked as @Retired will be removed after 1 minor release
> sharply
> >> > (using ArchUnit to force it, needs further check whether it is
> possible).
> >> > Devs who marked them as @Retired are responsible to remove them.
> >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> >> > recommended. @Experimental -> @Retired and @Internal -> @Retired could
> >> also
> >> > be used if it can increase user-friendliness or dev-friendliness, but
> not
> >> > mandatory.
> >> > 4. Some variables will be defined in @Retired to support the
> deprecation
> >> > process management. Further extension is possible, since the
> annotation
> >> is
> >> > built by us.
> >> >
> >> >
> >> > Best regards,
> >> > Jing
> >> >
> >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi Xintong,
> >> >>
> >> >> Thanks for the explanation. Please see the replies inline below.
> >> >>
> >> >> I agree. And from my understanding, demoting a Public API is also a
> >> kind of
> >> >> > such change, just like removing one, which can only happen with
> major
> >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> >> anytime,
> >> >> but
> >> >> > only in the case major version bumps happen before reaching the
> >> >> > 2-minor-release migration period. Actually, demoting would be a
> weaker
> >> >> > change compared to removing the API immediately upon major version
> >> bumps,
> >> >> > in order to keep the commitment about the 2-minor-release migration
> >> >> period.
> >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> against
> >> >> > conventions, we may introduce a new annotation if necessary, e.g.,
> >> >> > `@PublicRetiring`, to avoid confusions.
> >> >>
> >> >> As an end user who only uses Public APIs, if I don't change my code
> at
> >> all,
> >> >> my expectation is the following:
> >> >> 1. Upgrading from 1.x to 2.x may have issues.
> >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with
> all
> >> >> the 2.x versions.
> >> >> Actually I think there are some dependency version resolution
> policies
> >> out
> >> >> there which picks the highest minor version when the dependencies
> pull
> >> in
> >> >> multiple minor versions of the same jar, which may be broken if we
> >> remove
> >> >> the API in minor releases.
> >> >>
> >> >> I'm not sure about this. Yes, it's completely "legal" that we bump up
> >> the
> >> >> > major version whenever a breaking change is needed. However, this
> also
> >> >> > weakens the value of the commitment that public APIs will stay
> stable
> >> >> > within the major release series, as the series can end anytime.
> IMHO,
> >> >> short
> >> >> > major release series are not something "make the end users happy",
> but
> >> >> > backdoors that allow us as the developers to make frequent breaking
> >> >> > changes. On the contrary, with the demoting approach, we can still
> >> have
> >> >> > longer major release series, while only allowing Public APIs
> >> deprecated
> >> >> at
> >> >> > the end of the previous major version to be removed in the next
> major
> >> >> > version.
> >> >>
> >> >> I totally agree that frequent major version bumps are not ideal, but
> >> here
> >> >> we are comparing it with a minor version bump which removes a Public
> >> API.
> >> >> So the context is that we have already decided to remove this Public
> API
> >> >> while keeping everything else backwards compatible. I think a major
> >> version
> >> >> bump is a commonly understood signal here, compared with a minor
> version
> >> >> change. From end users' perspective, for those who are not impacted,
> in
> >> >> this case upgrading a major version is not necessarily more involved
> >> than
> >> >> upgrading a minor version - both should be as smooth as a dependency
> >> >> version change. For those who are impacted, they will lose the Public
> >> API
> >> >> anyways and a major version bump ensures there is no surprise.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jiangjie (Becket) Qin
> >> >>
> >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <tonysong820@gmail.com
> >
> >> >> wrote:
> >> >>
> >> >> > Public API is a well defined common concept, and one of its
> >> >> >> convention is that it only changes with a major version change.
> >> >> >>
> >> >> >
> >> >> > I agree. And from my understanding, demoting a Public API is also a
> >> kind
> >> >> > of such change, just like removing one, which can only happen with
> >> major
> >> >> > version bumps. I'm not proposing to allow demoting Public APIs
> >> anytime,
> >> >> but
> >> >> > only in the case major version bumps happen before reaching the
> >> >> > 2-minor-release migration period. Actually, demoting would be a
> weaker
> >> >> > change compared to removing the API immediately upon major version
> >> bumps,
> >> >> > in order to keep the commitment about the 2-minor-release migration
> >> >> period.
> >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds
> against
> >> >> > conventions, we may introduce a new annotation if necessary, e.g.,
> >> >> > `@PublicRetiring`, to avoid confusions.
> >> >> >
> >> >> > But it should be
> >> >> >> completely OK to bump up the major version if we really want to
> get
> >> rid
> >> >> of
> >> >> >> a public API, right?
> >> >> >>
> >> >> >
> >> >> > I'm not sure about this. Yes, it's completely "legal" that we bump
> up
> >> the
> >> >> > major version whenever a breaking change is needed. However, this
> also
> >> >> > weakens the value of the commitment that public APIs will stay
> stable
> >> >> > within the major release series, as the series can end anytime.
> IMHO,
> >> >> short
> >> >> > major release series are not something "make the end users happy",
> but
> >> >> > backdoors that allow us as the developers to make frequent breaking
> >> >> > changes. On the contrary, with the demoting approach, we can still
> >> have
> >> >> > longer major release series, while only allowing Public APIs
> >> deprecated
> >> >> at
> >> >> > the end of the previous major version to be removed in the next
> major
> >> >> > version.
> >> >> >
> >> >> > Given our track record I would prefer a regular cycle (1-2 years)
> to
> >> >> >> force us to think about this whole topic, and not put it again to
> the
> >> >> >> wayside and giving us (and users) a clear expectation on when
> >> breaking
> >> >> >> changes can be made.
> >> >> >>
> >> >> >
> >> >> > +1. I personally think 2-3 years would be a good time for new major
> >> >> > versions, or longer if there's no breaking changes needed. That
> makes
> >> 1-2
> >> >> > year a perfect time to revisit the topic, while leaving us more
> time
> >> to
> >> >> > prepare the major release if needed.
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > Xintong
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <
> chesnay@apache.org
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> >> >> > It would be valuable if we can avoid releasing minor versions
> for
> >> >> >> previous
> >> >> >> > major versions.
> >> >> >>
> >> >> >> On paper, /absolutely /agree, but I'm not sure how viable that is
> in
> >> >> >> practice.
> >> >> >>
> >> >> >> On the current 2.0 agenda is potentially dropping support for Java
> >> 8/11,
> >> >> >> which may very well be a problem for our current users.
> >> >> >>
> >> >> >>
> >> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> >> >> > Thanks for the feedback and sorry for the confusion about Public
> >> API
> >> >> >> > deprecation. I just noticed that there was a mistake in the
> NOTES
> >> part
> >> >> >> for
> >> >> >> > Public API due to a copy-paste error... I just fixed it.
> >> >> >> I'm very relieved to hear that. Glad to hear that we are on the
> same
> >> >> >> page on that note.
> >> >> >>
> >> >> >>
> >> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> >> >> >> > But it should be
> >> >> >> > completely OK to bump up the major version if we really want to
> get
> >> >> rid
> >> >> >> of
> >> >> >> > a public API, right?
> >> >> >>
> >> >> >> Technically yes, but look at how long it took to get us to 2.0. ;)
> >> >> >>
> >> >> >> There's a separate discussion to be had on the cadence of major
> >> releases
> >> >> >> going forward, and there seem to be different opinions on that.
> >> >> >>
> >> >> >> If we take the Kafka example of 2 minor releases between major
> ones,
> >> >> >> that for us means that users have to potentially deal with
> breaking
> >> >> >> changes every 6 months, which seems like a lot.
> >> >> >>
> >> >> >> Given our track record I would prefer a regular cycle (1-2 years)
> to
> >> >> >> force us to think about this whole topic, and not put it again to
> the
> >> >> >> wayside and giving us (and users) a clear expectation on when
> >> breaking
> >> >> >> changes can be made.
> >> >> >>
> >> >> >> But again, maybe this should be in a separate thread.
> >> >> >>
> >> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> >> >> >> > Do you have an example of behavioral change in mind? Not sure I
> >> fully
> >> >> >> > understand the concern for behavioral change here.
> >> >> >>
> >> >> >> This could be a lot of things. It can be performance in certain
> >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
> >> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> >> >> >>
> >> >> >> For a concrete example, consider the job submission. A few
> releases
> >> back
> >> >> >> we made changes such that the initialization of the job master
> >> happens
> >> >> >> asynchronously.
> >> >> >> This meant the job submission call returns sooner, and the job
> state
> >> >> >> enum was extended to cover this state.
> >> >> >> API-wise we consider this a compatible change, but the observed
> >> behavior
> >> >> >> may be different.
> >> >> >>
> >> >> >> Metrics are another example; I believe over time we changed what
> some
> >> >> >> metrics returned a few times.
> >> >> >>
> >> >> >
> >> >>
> >>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by John Roesler <vv...@apache.org>.
Hi Becket,

Thanks for the reply! I’d like to continue the conversation about compatibility outside of this FLIP thread, but for now, I can accept your decision. It’s certainly an improvement. 

Thanks again,
John

On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> Hi John,
>
> Completely agree with all you said.
>
> Can we consider only dropping deprecated APIs in major releases across the
>> board? I understand that Experimental and PublicEvolving APIs are by
>> definition less stable, but it seems like this should be reflected in the
>> required deprecation period alone. I.e. that we must keep them around for
>> at least zero or one minor release, not that we can drop them in a minor or
>> patch release.
>
> Personally speaking, I would love to do this, for exactly the reason you
> mentioned. However, I did not propose this due to the following reasons:
>
> 1. I am hesitating a little bit about changing the accepted FLIPs too soon.
> 2. More importantly, to avoid slowing down our development. At this point,
> Flink still lacks some design / routines to support good API evolvability /
> extensibility. Just like you said, it takes some time to be good at this.
> In this case, my concern is that only removing Experimental /
> PublicEvolving APIs in major version changes may result in too much
> overhead and dramatically slow down the development of Flink. So, I was
> thinking that we can start with the current status. Hopefully after we are
> more comfortable with the maintenance overhead of deprecated APIs, we can
> then have a stronger guarantee for Experimental / PublicEvolving APIs.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org> wrote:
>
>> Hi Becket,
>>
>> Thanks for this FLIP! Having a deprecation process is really important. I
>> understand some people’s concerns about the additional burden for project
>> maintainers, but my personal experience with Kafka has been that it’s very
>> liveable and that it’s well worth the benefit to users. In fact, users
>> being able to confidently upgrade is also a benefit to maintainers, as we
>> will get fewer questions from people stuck on very old versions.
>>
>> One question:
>> Can we consider only dropping deprecated APIs in major releases across the
>> board? I understand that Experimental and PublicEvolving APIs are by
>> definition less stable, but it seems like this should be reflected in the
>> required deprecation period alone. I.e. that we must keep them around for
>> at least zero or one minor release, not that we can drop them in a minor or
>> patch release.
>>
>> The advantage of forbidding the removal of any API in minor or patch
>> releases is that users will get a strong guarantee that they can bump the
>> minor or patch version and still be able to compile, or even just re-link
>> and know that they won’t face “MethodDef” exceptions at run time. This is a
>> binary guarantee: if we allow removing  even Experimental APIs outside of
>> major releases, users can no longer confidently upgrade.
>>
>> Aside from that, I’d share my 2 cents on a couple of points:
>> * I’d use the official Deprecated annotation instead of introducing our
>> own flavor (Retired, etc), since Deprecated is well integrated into build
>> tools and IDEs.
>> * I wouldn’t worry about a demotion process in this FLIP; it seems
>> orthogonal, and something that should probably be taken case-by-case
>> anyway.
>> * Aside from deprecation and removal, there have been some discussions
>> about how to evolve APIs and behavior in compatible ways. This is somewhat
>> of an art, and if folks haven’t wrestled with it before, it’ll take some
>> time to become good at it. I feel like this topic should also be orthogonal
>> to this FLIP, but FWIW, my suggestion would be to adopt a simple policy not
>> to break existing user programs, and leave the “how” up to implementers and
>> reviewers.
>>
>> Thanks again,
>> John
>>
>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
>> > Hi All,
>> >
>> > The @Public -> @PublicEvolving proposed by Xintong is a great idea.
>> > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
>> > minor release)--> @Public --> @deprecated --(1 major
>> > release)--> @PublicRetired. It will provide a lot of flexibility without
>> > breaking any rules we had. @Public APIs are allowed to change between
>> major
>> > releases. Changing annotations is acceptable and provides additional
>> > tolerance i.e. user-friendliness, since the APIs themself are not
>> changed.
>> >
>> > I had similar thoughts when I was facing those issues. I want to move one
>> > step further and suggest introducing one more annotation @Retired.
>> >
>> > Not like the @PublicRetired which is a compromise of downgrading @Public
>> to
>> > @PublicEvolving. As I mentioned earlier in my reply, Java standard
>> > @deprecated should be used in the early stage of the deprecation process
>> > and doesn't really meet our requirement. Since Java does not allow us to
>> > extend annotation, I think it would be feasible to have the new @Retired
>> to
>> > help us monitor and manage the deprecation process, house cleaning, etc.
>> >
>> > Some ideas could be(open for discussion):
>> >
>> > @Retired:
>> >
>> > 1. There must be a replacement with functionality compatibility before
>> APIs
>> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are
>> > mandatory (a FLIP is recommended).
>> > 2. APIs marked as @Retired will be removed after 1 minor release sharply
>> > (using ArchUnit to force it, needs further check whether it is possible).
>> > Devs who marked them as @Retired are responsible to remove them.
>> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
>> > recommended. @Experimental -> @Retired and @Internal -> @Retired could
>> also
>> > be used if it can increase user-friendliness or dev-friendliness, but not
>> > mandatory.
>> > 4. Some variables will be defined in @Retired to support the deprecation
>> > process management. Further extension is possible, since the annotation
>> is
>> > built by us.
>> >
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com>
>> wrote:
>> >
>> >> Hi Xintong,
>> >>
>> >> Thanks for the explanation. Please see the replies inline below.
>> >>
>> >> I agree. And from my understanding, demoting a Public API is also a
>> kind of
>> >> > such change, just like removing one, which can only happen with major
>> >> > version bumps. I'm not proposing to allow demoting Public APIs
>> anytime,
>> >> but
>> >> > only in the case major version bumps happen before reaching the
>> >> > 2-minor-release migration period. Actually, demoting would be a weaker
>> >> > change compared to removing the API immediately upon major version
>> bumps,
>> >> > in order to keep the commitment about the 2-minor-release migration
>> >> period.
>> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
>> >> > conventions, we may introduce a new annotation if necessary, e.g.,
>> >> > `@PublicRetiring`, to avoid confusions.
>> >>
>> >> As an end user who only uses Public APIs, if I don't change my code at
>> all,
>> >> my expectation is the following:
>> >> 1. Upgrading from 1.x to 2.x may have issues.
>> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
>> >> the 2.x versions.
>> >> Actually I think there are some dependency version resolution policies
>> out
>> >> there which picks the highest minor version when the dependencies pull
>> in
>> >> multiple minor versions of the same jar, which may be broken if we
>> remove
>> >> the API in minor releases.
>> >>
>> >> I'm not sure about this. Yes, it's completely "legal" that we bump up
>> the
>> >> > major version whenever a breaking change is needed. However, this also
>> >> > weakens the value of the commitment that public APIs will stay stable
>> >> > within the major release series, as the series can end anytime. IMHO,
>> >> short
>> >> > major release series are not something "make the end users happy", but
>> >> > backdoors that allow us as the developers to make frequent breaking
>> >> > changes. On the contrary, with the demoting approach, we can still
>> have
>> >> > longer major release series, while only allowing Public APIs
>> deprecated
>> >> at
>> >> > the end of the previous major version to be removed in the next major
>> >> > version.
>> >>
>> >> I totally agree that frequent major version bumps are not ideal, but
>> here
>> >> we are comparing it with a minor version bump which removes a Public
>> API.
>> >> So the context is that we have already decided to remove this Public API
>> >> while keeping everything else backwards compatible. I think a major
>> version
>> >> bump is a commonly understood signal here, compared with a minor version
>> >> change. From end users' perspective, for those who are not impacted, in
>> >> this case upgrading a major version is not necessarily more involved
>> than
>> >> upgrading a minor version - both should be as smooth as a dependency
>> >> version change. For those who are impacted, they will lose the Public
>> API
>> >> anyways and a major version bump ensures there is no surprise.
>> >>
>> >> Thanks,
>> >>
>> >> Jiangjie (Becket) Qin
>> >>
>> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com>
>> >> wrote:
>> >>
>> >> > Public API is a well defined common concept, and one of its
>> >> >> convention is that it only changes with a major version change.
>> >> >>
>> >> >
>> >> > I agree. And from my understanding, demoting a Public API is also a
>> kind
>> >> > of such change, just like removing one, which can only happen with
>> major
>> >> > version bumps. I'm not proposing to allow demoting Public APIs
>> anytime,
>> >> but
>> >> > only in the case major version bumps happen before reaching the
>> >> > 2-minor-release migration period. Actually, demoting would be a weaker
>> >> > change compared to removing the API immediately upon major version
>> bumps,
>> >> > in order to keep the commitment about the 2-minor-release migration
>> >> period.
>> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
>> >> > conventions, we may introduce a new annotation if necessary, e.g.,
>> >> > `@PublicRetiring`, to avoid confusions.
>> >> >
>> >> > But it should be
>> >> >> completely OK to bump up the major version if we really want to get
>> rid
>> >> of
>> >> >> a public API, right?
>> >> >>
>> >> >
>> >> > I'm not sure about this. Yes, it's completely "legal" that we bump up
>> the
>> >> > major version whenever a breaking change is needed. However, this also
>> >> > weakens the value of the commitment that public APIs will stay stable
>> >> > within the major release series, as the series can end anytime. IMHO,
>> >> short
>> >> > major release series are not something "make the end users happy", but
>> >> > backdoors that allow us as the developers to make frequent breaking
>> >> > changes. On the contrary, with the demoting approach, we can still
>> have
>> >> > longer major release series, while only allowing Public APIs
>> deprecated
>> >> at
>> >> > the end of the previous major version to be removed in the next major
>> >> > version.
>> >> >
>> >> > Given our track record I would prefer a regular cycle (1-2 years) to
>> >> >> force us to think about this whole topic, and not put it again to the
>> >> >> wayside and giving us (and users) a clear expectation on when
>> breaking
>> >> >> changes can be made.
>> >> >>
>> >> >
>> >> > +1. I personally think 2-3 years would be a good time for new major
>> >> > versions, or longer if there's no breaking changes needed. That makes
>> 1-2
>> >> > year a perfect time to revisit the topic, while leaving us more time
>> to
>> >> > prepare the major release if needed.
>> >> >
>> >> > Best,
>> >> >
>> >> > Xintong
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <chesnay@apache.org
>> >
>> >> > wrote:
>> >> >
>> >> >> On 13/06/2023 17:26, Becket Qin wrote:
>> >> >> > It would be valuable if we can avoid releasing minor versions for
>> >> >> previous
>> >> >> > major versions.
>> >> >>
>> >> >> On paper, /absolutely /agree, but I'm not sure how viable that is in
>> >> >> practice.
>> >> >>
>> >> >> On the current 2.0 agenda is potentially dropping support for Java
>> 8/11,
>> >> >> which may very well be a problem for our current users.
>> >> >>
>> >> >>
>> >> >> On 13/06/2023 17:26, Becket Qin wrote:
>> >> >> > Thanks for the feedback and sorry for the confusion about Public
>> API
>> >> >> > deprecation. I just noticed that there was a mistake in the NOTES
>> part
>> >> >> for
>> >> >> > Public API due to a copy-paste error... I just fixed it.
>> >> >> I'm very relieved to hear that. Glad to hear that we are on the same
>> >> >> page on that note.
>> >> >>
>> >> >>
>> >> >> On 15/06/2023 15:20, Becket Qin wrote:
>> >> >> > But it should be
>> >> >> > completely OK to bump up the major version if we really want to get
>> >> rid
>> >> >> of
>> >> >> > a public API, right?
>> >> >>
>> >> >> Technically yes, but look at how long it took to get us to 2.0. ;)
>> >> >>
>> >> >> There's a separate discussion to be had on the cadence of major
>> releases
>> >> >> going forward, and there seem to be different opinions on that.
>> >> >>
>> >> >> If we take the Kafka example of 2 minor releases between major ones,
>> >> >> that for us means that users have to potentially deal with breaking
>> >> >> changes every 6 months, which seems like a lot.
>> >> >>
>> >> >> Given our track record I would prefer a regular cycle (1-2 years) to
>> >> >> force us to think about this whole topic, and not put it again to the
>> >> >> wayside and giving us (and users) a clear expectation on when
>> breaking
>> >> >> changes can be made.
>> >> >>
>> >> >> But again, maybe this should be in a separate thread.
>> >> >>
>> >> >> On 14/06/2023 11:37, Becket Qin wrote:
>> >> >> > Do you have an example of behavioral change in mind? Not sure I
>> fully
>> >> >> > understand the concern for behavioral change here.
>> >> >>
>> >> >> This could be a lot of things. It can be performance in certain
>> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
>> >> >> (https://xkcd.com/1172/), a semantic change to some API.
>> >> >>
>> >> >> For a concrete example, consider the job submission. A few releases
>> back
>> >> >> we made changes such that the initialization of the job master
>> happens
>> >> >> asynchronously.
>> >> >> This meant the job submission call returns sooner, and the job state
>> >> >> enum was extended to cover this state.
>> >> >> API-wise we consider this a compatible change, but the observed
>> behavior
>> >> >> may be different.
>> >> >>
>> >> >> Metrics are another example; I believe over time we changed what some
>> >> >> metrics returned a few times.
>> >> >>
>> >> >
>> >>
>>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi John,

Completely agree with all you said.

Can we consider only dropping deprecated APIs in major releases across the
> board? I understand that Experimental and PublicEvolving APIs are by
> definition less stable, but it seems like this should be reflected in the
> required deprecation period alone. I.e. that we must keep them around for
> at least zero or one minor release, not that we can drop them in a minor or
> patch release.

Personally speaking, I would love to do this, for exactly the reason you
mentioned. However, I did not propose this due to the following reasons:

1. I am hesitating a little bit about changing the accepted FLIPs too soon.
2. More importantly, to avoid slowing down our development. At this point,
Flink still lacks some design / routines to support good API evolvability /
extensibility. Just like you said, it takes some time to be good at this.
In this case, my concern is that only removing Experimental /
PublicEvolving APIs in major version changes may result in too much
overhead and dramatically slow down the development of Flink. So, I was
thinking that we can start with the current status. Hopefully after we are
more comfortable with the maintenance overhead of deprecated APIs, we can
then have a stronger guarantee for Experimental / PublicEvolving APIs.

Thanks,

Jiangjie (Becket) Qin



On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vv...@apache.org> wrote:

> Hi Becket,
>
> Thanks for this FLIP! Having a deprecation process is really important. I
> understand some people’s concerns about the additional burden for project
> maintainers, but my personal experience with Kafka has been that it’s very
> liveable and that it’s well worth the benefit to users. In fact, users
> being able to confidently upgrade is also a benefit to maintainers, as we
> will get fewer questions from people stuck on very old versions.
>
> One question:
> Can we consider only dropping deprecated APIs in major releases across the
> board? I understand that Experimental and PublicEvolving APIs are by
> definition less stable, but it seems like this should be reflected in the
> required deprecation period alone. I.e. that we must keep them around for
> at least zero or one minor release, not that we can drop them in a minor or
> patch release.
>
> The advantage of forbidding the removal of any API in minor or patch
> releases is that users will get a strong guarantee that they can bump the
> minor or patch version and still be able to compile, or even just re-link
> and know that they won’t face “MethodDef” exceptions at run time. This is a
> binary guarantee: if we allow removing  even Experimental APIs outside of
> major releases, users can no longer confidently upgrade.
>
> Aside from that, I’d share my 2 cents on a couple of points:
> * I’d use the official Deprecated annotation instead of introducing our
> own flavor (Retired, etc), since Deprecated is well integrated into build
> tools and IDEs.
> * I wouldn’t worry about a demotion process in this FLIP; it seems
> orthogonal, and something that should probably be taken case-by-case
> anyway.
> * Aside from deprecation and removal, there have been some discussions
> about how to evolve APIs and behavior in compatible ways. This is somewhat
> of an art, and if folks haven’t wrestled with it before, it’ll take some
> time to become good at it. I feel like this topic should also be orthogonal
> to this FLIP, but FWIW, my suggestion would be to adopt a simple policy not
> to break existing user programs, and leave the “how” up to implementers and
> reviewers.
>
> Thanks again,
> John
>
> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > Hi All,
> >
> > The @Public -> @PublicEvolving proposed by Xintong is a great idea.
> > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
> > minor release)--> @Public --> @deprecated --(1 major
> > release)--> @PublicRetired. It will provide a lot of flexibility without
> > breaking any rules we had. @Public APIs are allowed to change between
> major
> > releases. Changing annotations is acceptable and provides additional
> > tolerance i.e. user-friendliness, since the APIs themself are not
> changed.
> >
> > I had similar thoughts when I was facing those issues. I want to move one
> > step further and suggest introducing one more annotation @Retired.
> >
> > Not like the @PublicRetired which is a compromise of downgrading @Public
> to
> > @PublicEvolving. As I mentioned earlier in my reply, Java standard
> > @deprecated should be used in the early stage of the deprecation process
> > and doesn't really meet our requirement. Since Java does not allow us to
> > extend annotation, I think it would be feasible to have the new @Retired
> to
> > help us monitor and manage the deprecation process, house cleaning, etc.
> >
> > Some ideas could be(open for discussion):
> >
> > @Retired:
> >
> > 1. There must be a replacement with functionality compatibility before
> APIs
> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are
> > mandatory (a FLIP is recommended).
> > 2. APIs marked as @Retired will be removed after 1 minor release sharply
> > (using ArchUnit to force it, needs further check whether it is possible).
> > Devs who marked them as @Retired are responsible to remove them.
> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> > recommended. @Experimental -> @Retired and @Internal -> @Retired could
> also
> > be used if it can increase user-friendliness or dev-friendliness, but not
> > mandatory.
> > 4. Some variables will be defined in @Retired to support the deprecation
> > process management. Further extension is possible, since the annotation
> is
> > built by us.
> >
> >
> > Best regards,
> > Jing
> >
> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com>
> wrote:
> >
> >> Hi Xintong,
> >>
> >> Thanks for the explanation. Please see the replies inline below.
> >>
> >> I agree. And from my understanding, demoting a Public API is also a
> kind of
> >> > such change, just like removing one, which can only happen with major
> >> > version bumps. I'm not proposing to allow demoting Public APIs
> anytime,
> >> but
> >> > only in the case major version bumps happen before reaching the
> >> > 2-minor-release migration period. Actually, demoting would be a weaker
> >> > change compared to removing the API immediately upon major version
> bumps,
> >> > in order to keep the commitment about the 2-minor-release migration
> >> period.
> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
> >> > conventions, we may introduce a new annotation if necessary, e.g.,
> >> > `@PublicRetiring`, to avoid confusions.
> >>
> >> As an end user who only uses Public APIs, if I don't change my code at
> all,
> >> my expectation is the following:
> >> 1. Upgrading from 1.x to 2.x may have issues.
> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
> >> the 2.x versions.
> >> Actually I think there are some dependency version resolution policies
> out
> >> there which picks the highest minor version when the dependencies pull
> in
> >> multiple minor versions of the same jar, which may be broken if we
> remove
> >> the API in minor releases.
> >>
> >> I'm not sure about this. Yes, it's completely "legal" that we bump up
> the
> >> > major version whenever a breaking change is needed. However, this also
> >> > weakens the value of the commitment that public APIs will stay stable
> >> > within the major release series, as the series can end anytime. IMHO,
> >> short
> >> > major release series are not something "make the end users happy", but
> >> > backdoors that allow us as the developers to make frequent breaking
> >> > changes. On the contrary, with the demoting approach, we can still
> have
> >> > longer major release series, while only allowing Public APIs
> deprecated
> >> at
> >> > the end of the previous major version to be removed in the next major
> >> > version.
> >>
> >> I totally agree that frequent major version bumps are not ideal, but
> here
> >> we are comparing it with a minor version bump which removes a Public
> API.
> >> So the context is that we have already decided to remove this Public API
> >> while keeping everything else backwards compatible. I think a major
> version
> >> bump is a commonly understood signal here, compared with a minor version
> >> change. From end users' perspective, for those who are not impacted, in
> >> this case upgrading a major version is not necessarily more involved
> than
> >> upgrading a minor version - both should be as smooth as a dependency
> >> version change. For those who are impacted, they will lose the Public
> API
> >> anyways and a major version bump ensures there is no surprise.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com>
> >> wrote:
> >>
> >> > Public API is a well defined common concept, and one of its
> >> >> convention is that it only changes with a major version change.
> >> >>
> >> >
> >> > I agree. And from my understanding, demoting a Public API is also a
> kind
> >> > of such change, just like removing one, which can only happen with
> major
> >> > version bumps. I'm not proposing to allow demoting Public APIs
> anytime,
> >> but
> >> > only in the case major version bumps happen before reaching the
> >> > 2-minor-release migration period. Actually, demoting would be a weaker
> >> > change compared to removing the API immediately upon major version
> bumps,
> >> > in order to keep the commitment about the 2-minor-release migration
> >> period.
> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
> >> > conventions, we may introduce a new annotation if necessary, e.g.,
> >> > `@PublicRetiring`, to avoid confusions.
> >> >
> >> > But it should be
> >> >> completely OK to bump up the major version if we really want to get
> rid
> >> of
> >> >> a public API, right?
> >> >>
> >> >
> >> > I'm not sure about this. Yes, it's completely "legal" that we bump up
> the
> >> > major version whenever a breaking change is needed. However, this also
> >> > weakens the value of the commitment that public APIs will stay stable
> >> > within the major release series, as the series can end anytime. IMHO,
> >> short
> >> > major release series are not something "make the end users happy", but
> >> > backdoors that allow us as the developers to make frequent breaking
> >> > changes. On the contrary, with the demoting approach, we can still
> have
> >> > longer major release series, while only allowing Public APIs
> deprecated
> >> at
> >> > the end of the previous major version to be removed in the next major
> >> > version.
> >> >
> >> > Given our track record I would prefer a regular cycle (1-2 years) to
> >> >> force us to think about this whole topic, and not put it again to the
> >> >> wayside and giving us (and users) a clear expectation on when
> breaking
> >> >> changes can be made.
> >> >>
> >> >
> >> > +1. I personally think 2-3 years would be a good time for new major
> >> > versions, or longer if there's no breaking changes needed. That makes
> 1-2
> >> > year a perfect time to revisit the topic, while leaving us more time
> to
> >> > prepare the major release if needed.
> >> >
> >> > Best,
> >> >
> >> > Xintong
> >> >
> >> >
> >> >
> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <chesnay@apache.org
> >
> >> > wrote:
> >> >
> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> >> > It would be valuable if we can avoid releasing minor versions for
> >> >> previous
> >> >> > major versions.
> >> >>
> >> >> On paper, /absolutely /agree, but I'm not sure how viable that is in
> >> >> practice.
> >> >>
> >> >> On the current 2.0 agenda is potentially dropping support for Java
> 8/11,
> >> >> which may very well be a problem for our current users.
> >> >>
> >> >>
> >> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> >> > Thanks for the feedback and sorry for the confusion about Public
> API
> >> >> > deprecation. I just noticed that there was a mistake in the NOTES
> part
> >> >> for
> >> >> > Public API due to a copy-paste error... I just fixed it.
> >> >> I'm very relieved to hear that. Glad to hear that we are on the same
> >> >> page on that note.
> >> >>
> >> >>
> >> >> On 15/06/2023 15:20, Becket Qin wrote:
> >> >> > But it should be
> >> >> > completely OK to bump up the major version if we really want to get
> >> rid
> >> >> of
> >> >> > a public API, right?
> >> >>
> >> >> Technically yes, but look at how long it took to get us to 2.0. ;)
> >> >>
> >> >> There's a separate discussion to be had on the cadence of major
> releases
> >> >> going forward, and there seem to be different opinions on that.
> >> >>
> >> >> If we take the Kafka example of 2 minor releases between major ones,
> >> >> that for us means that users have to potentially deal with breaking
> >> >> changes every 6 months, which seems like a lot.
> >> >>
> >> >> Given our track record I would prefer a regular cycle (1-2 years) to
> >> >> force us to think about this whole topic, and not put it again to the
> >> >> wayside and giving us (and users) a clear expectation on when
> breaking
> >> >> changes can be made.
> >> >>
> >> >> But again, maybe this should be in a separate thread.
> >> >>
> >> >> On 14/06/2023 11:37, Becket Qin wrote:
> >> >> > Do you have an example of behavioral change in mind? Not sure I
> fully
> >> >> > understand the concern for behavioral change here.
> >> >>
> >> >> This could be a lot of things. It can be performance in certain
> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
> >> >> (https://xkcd.com/1172/), a semantic change to some API.
> >> >>
> >> >> For a concrete example, consider the job submission. A few releases
> back
> >> >> we made changes such that the initialization of the job master
> happens
> >> >> asynchronously.
> >> >> This meant the job submission call returns sooner, and the job state
> >> >> enum was extended to cover this state.
> >> >> API-wise we consider this a compatible change, but the observed
> behavior
> >> >> may be different.
> >> >>
> >> >> Metrics are another example; I believe over time we changed what some
> >> >> metrics returned a few times.
> >> >>
> >> >
> >>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by John Roesler <vv...@apache.org>.
Hi Becket,

Thanks for this FLIP! Having a deprecation process is really important. I understand some people’s concerns about the additional burden for project maintainers, but my personal experience with Kafka has been that it’s very liveable and that it’s well worth the benefit to users. In fact, users being able to confidently upgrade is also a benefit to maintainers, as we will get fewer questions from people stuck on very old versions.

One question:
Can we consider only dropping deprecated APIs in major releases across the board? I understand that Experimental and PublicEvolving APIs are by definition less stable, but it seems like this should be reflected in the required deprecation period alone. I.e. that we must keep them around for at least zero or one minor release, not that we can drop them in a minor or patch release. 

The advantage of forbidding the removal of any API in minor or patch releases is that users will get a strong guarantee that they can bump the minor or patch version and still be able to compile, or even just re-link and know that they won’t face “MethodDef” exceptions at run time. This is a binary guarantee: if we allow removing  even Experimental APIs outside of major releases, users can no longer confidently upgrade.

Aside from that, I’d share my 2 cents on a couple of points:
* I’d use the official Deprecated annotation instead of introducing our own flavor (Retired, etc), since Deprecated is well integrated into build tools and IDEs.
* I wouldn’t worry about a demotion process in this FLIP; it seems orthogonal, and something that should probably be taken case-by-case anyway. 
* Aside from deprecation and removal, there have been some discussions about how to evolve APIs and behavior in compatible ways. This is somewhat of an art, and if folks haven’t wrestled with it before, it’ll take some time to become good at it. I feel like this topic should also be orthogonal to this FLIP, but FWIW, my suggestion would be to adopt a simple policy not to break existing user programs, and leave the “how” up to implementers and reviewers. 

Thanks again,
John

On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> Hi All,
>
> The @Public -> @PublicEvolving proposed by Xintong is a great idea.
> Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
> minor release)--> @Public --> @deprecated --(1 major
> release)--> @PublicRetired. It will provide a lot of flexibility without
> breaking any rules we had. @Public APIs are allowed to change between major
> releases. Changing annotations is acceptable and provides additional
> tolerance i.e. user-friendliness, since the APIs themself are not changed.
>
> I had similar thoughts when I was facing those issues. I want to move one
> step further and suggest introducing one more annotation @Retired.
>
> Not like the @PublicRetired which is a compromise of downgrading @Public to
> @PublicEvolving. As I mentioned earlier in my reply, Java standard
> @deprecated should be used in the early stage of the deprecation process
> and doesn't really meet our requirement. Since Java does not allow us to
> extend annotation, I think it would be feasible to have the new @Retired to
> help us monitor and manage the deprecation process, house cleaning, etc.
>
> Some ideas could be(open for discussion):
>
> @Retired:
>
> 1. There must be a replacement with functionality compatibility before APIs
> can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are
> mandatory (a FLIP is recommended).
> 2. APIs marked as @Retired will be removed after 1 minor release sharply
> (using ArchUnit to force it, needs further check whether it is possible).
> Devs who marked them as @Retired are responsible to remove them.
> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
> recommended. @Experimental -> @Retired and @Internal -> @Retired could also
> be used if it can increase user-friendliness or dev-friendliness, but not
> mandatory.
> 4. Some variables will be defined in @Retired to support the deprecation
> process management. Further extension is possible, since the annotation is
> built by us.
>
>
> Best regards,
> Jing
>
> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com> wrote:
>
>> Hi Xintong,
>>
>> Thanks for the explanation. Please see the replies inline below.
>>
>> I agree. And from my understanding, demoting a Public API is also a kind of
>> > such change, just like removing one, which can only happen with major
>> > version bumps. I'm not proposing to allow demoting Public APIs anytime,
>> but
>> > only in the case major version bumps happen before reaching the
>> > 2-minor-release migration period. Actually, demoting would be a weaker
>> > change compared to removing the API immediately upon major version bumps,
>> > in order to keep the commitment about the 2-minor-release migration
>> period.
>> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
>> > conventions, we may introduce a new annotation if necessary, e.g.,
>> > `@PublicRetiring`, to avoid confusions.
>>
>> As an end user who only uses Public APIs, if I don't change my code at all,
>> my expectation is the following:
>> 1. Upgrading from 1.x to 2.x may have issues.
>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
>> the 2.x versions.
>> Actually I think there are some dependency version resolution policies out
>> there which picks the highest minor version when the dependencies pull in
>> multiple minor versions of the same jar, which may be broken if we remove
>> the API in minor releases.
>>
>> I'm not sure about this. Yes, it's completely "legal" that we bump up the
>> > major version whenever a breaking change is needed. However, this also
>> > weakens the value of the commitment that public APIs will stay stable
>> > within the major release series, as the series can end anytime. IMHO,
>> short
>> > major release series are not something "make the end users happy", but
>> > backdoors that allow us as the developers to make frequent breaking
>> > changes. On the contrary, with the demoting approach, we can still have
>> > longer major release series, while only allowing Public APIs deprecated
>> at
>> > the end of the previous major version to be removed in the next major
>> > version.
>>
>> I totally agree that frequent major version bumps are not ideal, but here
>> we are comparing it with a minor version bump which removes a Public API.
>> So the context is that we have already decided to remove this Public API
>> while keeping everything else backwards compatible. I think a major version
>> bump is a commonly understood signal here, compared with a minor version
>> change. From end users' perspective, for those who are not impacted, in
>> this case upgrading a major version is not necessarily more involved than
>> upgrading a minor version - both should be as smooth as a dependency
>> version change. For those who are impacted, they will lose the Public API
>> anyways and a major version bump ensures there is no surprise.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com>
>> wrote:
>>
>> > Public API is a well defined common concept, and one of its
>> >> convention is that it only changes with a major version change.
>> >>
>> >
>> > I agree. And from my understanding, demoting a Public API is also a kind
>> > of such change, just like removing one, which can only happen with major
>> > version bumps. I'm not proposing to allow demoting Public APIs anytime,
>> but
>> > only in the case major version bumps happen before reaching the
>> > 2-minor-release migration period. Actually, demoting would be a weaker
>> > change compared to removing the API immediately upon major version bumps,
>> > in order to keep the commitment about the 2-minor-release migration
>> period.
>> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
>> > conventions, we may introduce a new annotation if necessary, e.g.,
>> > `@PublicRetiring`, to avoid confusions.
>> >
>> > But it should be
>> >> completely OK to bump up the major version if we really want to get rid
>> of
>> >> a public API, right?
>> >>
>> >
>> > I'm not sure about this. Yes, it's completely "legal" that we bump up the
>> > major version whenever a breaking change is needed. However, this also
>> > weakens the value of the commitment that public APIs will stay stable
>> > within the major release series, as the series can end anytime. IMHO,
>> short
>> > major release series are not something "make the end users happy", but
>> > backdoors that allow us as the developers to make frequent breaking
>> > changes. On the contrary, with the demoting approach, we can still have
>> > longer major release series, while only allowing Public APIs deprecated
>> at
>> > the end of the previous major version to be removed in the next major
>> > version.
>> >
>> > Given our track record I would prefer a regular cycle (1-2 years) to
>> >> force us to think about this whole topic, and not put it again to the
>> >> wayside and giving us (and users) a clear expectation on when breaking
>> >> changes can be made.
>> >>
>> >
>> > +1. I personally think 2-3 years would be a good time for new major
>> > versions, or longer if there's no breaking changes needed. That makes 1-2
>> > year a perfect time to revisit the topic, while leaving us more time to
>> > prepare the major release if needed.
>> >
>> > Best,
>> >
>> > Xintong
>> >
>> >
>> >
>> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ch...@apache.org>
>> > wrote:
>> >
>> >> On 13/06/2023 17:26, Becket Qin wrote:
>> >> > It would be valuable if we can avoid releasing minor versions for
>> >> previous
>> >> > major versions.
>> >>
>> >> On paper, /absolutely /agree, but I'm not sure how viable that is in
>> >> practice.
>> >>
>> >> On the current 2.0 agenda is potentially dropping support for Java 8/11,
>> >> which may very well be a problem for our current users.
>> >>
>> >>
>> >> On 13/06/2023 17:26, Becket Qin wrote:
>> >> > Thanks for the feedback and sorry for the confusion about Public API
>> >> > deprecation. I just noticed that there was a mistake in the NOTES part
>> >> for
>> >> > Public API due to a copy-paste error... I just fixed it.
>> >> I'm very relieved to hear that. Glad to hear that we are on the same
>> >> page on that note.
>> >>
>> >>
>> >> On 15/06/2023 15:20, Becket Qin wrote:
>> >> > But it should be
>> >> > completely OK to bump up the major version if we really want to get
>> rid
>> >> of
>> >> > a public API, right?
>> >>
>> >> Technically yes, but look at how long it took to get us to 2.0. ;)
>> >>
>> >> There's a separate discussion to be had on the cadence of major releases
>> >> going forward, and there seem to be different opinions on that.
>> >>
>> >> If we take the Kafka example of 2 minor releases between major ones,
>> >> that for us means that users have to potentially deal with breaking
>> >> changes every 6 months, which seems like a lot.
>> >>
>> >> Given our track record I would prefer a regular cycle (1-2 years) to
>> >> force us to think about this whole topic, and not put it again to the
>> >> wayside and giving us (and users) a clear expectation on when breaking
>> >> changes can be made.
>> >>
>> >> But again, maybe this should be in a separate thread.
>> >>
>> >> On 14/06/2023 11:37, Becket Qin wrote:
>> >> > Do you have an example of behavioral change in mind? Not sure I fully
>> >> > understand the concern for behavioral change here.
>> >>
>> >> This could be a lot of things. It can be performance in certain
>> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
>> >> (https://xkcd.com/1172/), a semantic change to some API.
>> >>
>> >> For a concrete example, consider the job submission. A few releases back
>> >> we made changes such that the initialization of the job master happens
>> >> asynchronously.
>> >> This meant the job submission call returns sooner, and the job state
>> >> enum was extended to cover this state.
>> >> API-wise we consider this a compatible change, but the observed behavior
>> >> may be different.
>> >>
>> >> Metrics are another example; I believe over time we changed what some
>> >> metrics returned a few times.
>> >>
>> >
>>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi All,

The @Public -> @PublicEvolving proposed by Xintong is a great idea.
Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2
minor release)--> @Public --> @deprecated --(1 major
release)--> @PublicRetired. It will provide a lot of flexibility without
breaking any rules we had. @Public APIs are allowed to change between major
releases. Changing annotations is acceptable and provides additional
tolerance i.e. user-friendliness, since the APIs themself are not changed.

I had similar thoughts when I was facing those issues. I want to move one
step further and suggest introducing one more annotation @Retired.

Not like the @PublicRetired which is a compromise of downgrading @Public to
@PublicEvolving. As I mentioned earlier in my reply, Java standard
@deprecated should be used in the early stage of the deprecation process
and doesn't really meet our requirement. Since Java does not allow us to
extend annotation, I think it would be feasible to have the new @Retired to
help us monitor and manage the deprecation process, house cleaning, etc.

Some ideas could be(open for discussion):

@Retired:

1. There must be a replacement with functionality compatibility before APIs
can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are
mandatory (a FLIP is recommended).
2. APIs marked as @Retired will be removed after 1 minor release sharply
(using ArchUnit to force it, needs further check whether it is possible).
Devs who marked them as @Retired are responsible to remove them.
3. Both @Public -> @Retired and @PublicEvolving -> @Retired are
recommended. @Experimental -> @Retired and @Internal -> @Retired could also
be used if it can increase user-friendliness or dev-friendliness, but not
mandatory.
4. Some variables will be defined in @Retired to support the deprecation
process management. Further extension is possible, since the annotation is
built by us.


Best regards,
Jing

On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <be...@gmail.com> wrote:

> Hi Xintong,
>
> Thanks for the explanation. Please see the replies inline below.
>
> I agree. And from my understanding, demoting a Public API is also a kind of
> > such change, just like removing one, which can only happen with major
> > version bumps. I'm not proposing to allow demoting Public APIs anytime,
> but
> > only in the case major version bumps happen before reaching the
> > 2-minor-release migration period. Actually, demoting would be a weaker
> > change compared to removing the API immediately upon major version bumps,
> > in order to keep the commitment about the 2-minor-release migration
> period.
> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
> > conventions, we may introduce a new annotation if necessary, e.g.,
> > `@PublicRetiring`, to avoid confusions.
>
> As an end user who only uses Public APIs, if I don't change my code at all,
> my expectation is the following:
> 1. Upgrading from 1.x to 2.x may have issues.
> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
> the 2.x versions.
> Actually I think there are some dependency version resolution policies out
> there which picks the highest minor version when the dependencies pull in
> multiple minor versions of the same jar, which may be broken if we remove
> the API in minor releases.
>
> I'm not sure about this. Yes, it's completely "legal" that we bump up the
> > major version whenever a breaking change is needed. However, this also
> > weakens the value of the commitment that public APIs will stay stable
> > within the major release series, as the series can end anytime. IMHO,
> short
> > major release series are not something "make the end users happy", but
> > backdoors that allow us as the developers to make frequent breaking
> > changes. On the contrary, with the demoting approach, we can still have
> > longer major release series, while only allowing Public APIs deprecated
> at
> > the end of the previous major version to be removed in the next major
> > version.
>
> I totally agree that frequent major version bumps are not ideal, but here
> we are comparing it with a minor version bump which removes a Public API.
> So the context is that we have already decided to remove this Public API
> while keeping everything else backwards compatible. I think a major version
> bump is a commonly understood signal here, compared with a minor version
> change. From end users' perspective, for those who are not impacted, in
> this case upgrading a major version is not necessarily more involved than
> upgrading a minor version - both should be as smooth as a dependency
> version change. For those who are impacted, they will lose the Public API
> anyways and a major version bump ensures there is no surprise.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Public API is a well defined common concept, and one of its
> >> convention is that it only changes with a major version change.
> >>
> >
> > I agree. And from my understanding, demoting a Public API is also a kind
> > of such change, just like removing one, which can only happen with major
> > version bumps. I'm not proposing to allow demoting Public APIs anytime,
> but
> > only in the case major version bumps happen before reaching the
> > 2-minor-release migration period. Actually, demoting would be a weaker
> > change compared to removing the API immediately upon major version bumps,
> > in order to keep the commitment about the 2-minor-release migration
> period.
> > If the concern is that `@Public` -> `@PublicEvolving` sounds against
> > conventions, we may introduce a new annotation if necessary, e.g.,
> > `@PublicRetiring`, to avoid confusions.
> >
> > But it should be
> >> completely OK to bump up the major version if we really want to get rid
> of
> >> a public API, right?
> >>
> >
> > I'm not sure about this. Yes, it's completely "legal" that we bump up the
> > major version whenever a breaking change is needed. However, this also
> > weakens the value of the commitment that public APIs will stay stable
> > within the major release series, as the series can end anytime. IMHO,
> short
> > major release series are not something "make the end users happy", but
> > backdoors that allow us as the developers to make frequent breaking
> > changes. On the contrary, with the demoting approach, we can still have
> > longer major release series, while only allowing Public APIs deprecated
> at
> > the end of the previous major version to be removed in the next major
> > version.
> >
> > Given our track record I would prefer a regular cycle (1-2 years) to
> >> force us to think about this whole topic, and not put it again to the
> >> wayside and giving us (and users) a clear expectation on when breaking
> >> changes can be made.
> >>
> >
> > +1. I personally think 2-3 years would be a good time for new major
> > versions, or longer if there's no breaking changes needed. That makes 1-2
> > year a perfect time to revisit the topic, while leaving us more time to
> > prepare the major release if needed.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> > It would be valuable if we can avoid releasing minor versions for
> >> previous
> >> > major versions.
> >>
> >> On paper, /absolutely /agree, but I'm not sure how viable that is in
> >> practice.
> >>
> >> On the current 2.0 agenda is potentially dropping support for Java 8/11,
> >> which may very well be a problem for our current users.
> >>
> >>
> >> On 13/06/2023 17:26, Becket Qin wrote:
> >> > Thanks for the feedback and sorry for the confusion about Public API
> >> > deprecation. I just noticed that there was a mistake in the NOTES part
> >> for
> >> > Public API due to a copy-paste error... I just fixed it.
> >> I'm very relieved to hear that. Glad to hear that we are on the same
> >> page on that note.
> >>
> >>
> >> On 15/06/2023 15:20, Becket Qin wrote:
> >> > But it should be
> >> > completely OK to bump up the major version if we really want to get
> rid
> >> of
> >> > a public API, right?
> >>
> >> Technically yes, but look at how long it took to get us to 2.0. ;)
> >>
> >> There's a separate discussion to be had on the cadence of major releases
> >> going forward, and there seem to be different opinions on that.
> >>
> >> If we take the Kafka example of 2 minor releases between major ones,
> >> that for us means that users have to potentially deal with breaking
> >> changes every 6 months, which seems like a lot.
> >>
> >> Given our track record I would prefer a regular cycle (1-2 years) to
> >> force us to think about this whole topic, and not put it again to the
> >> wayside and giving us (and users) a clear expectation on when breaking
> >> changes can be made.
> >>
> >> But again, maybe this should be in a separate thread.
> >>
> >> On 14/06/2023 11:37, Becket Qin wrote:
> >> > Do you have an example of behavioral change in mind? Not sure I fully
> >> > understand the concern for behavioral change here.
> >>
> >> This could be a lot of things. It can be performance in certain
> >> edge-cases, a bug fix that users (maybe unknowingly) relied upon
> >> (https://xkcd.com/1172/), a semantic change to some API.
> >>
> >> For a concrete example, consider the job submission. A few releases back
> >> we made changes such that the initialization of the job master happens
> >> asynchronously.
> >> This meant the job submission call returns sooner, and the job state
> >> enum was extended to cover this state.
> >> API-wise we consider this a compatible change, but the observed behavior
> >> may be different.
> >>
> >> Metrics are another example; I believe over time we changed what some
> >> metrics returned a few times.
> >>
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

Thanks for the explanation. Please see the replies inline below.

I agree. And from my understanding, demoting a Public API is also a kind of
> such change, just like removing one, which can only happen with major
> version bumps. I'm not proposing to allow demoting Public APIs anytime, but
> only in the case major version bumps happen before reaching the
> 2-minor-release migration period. Actually, demoting would be a weaker
> change compared to removing the API immediately upon major version bumps,
> in order to keep the commitment about the 2-minor-release migration period.
> If the concern is that `@Public` -> `@PublicEvolving` sounds against
> conventions, we may introduce a new annotation if necessary, e.g.,
> `@PublicRetiring`, to avoid confusions.

As an end user who only uses Public APIs, if I don't change my code at all,
my expectation is the following:
1. Upgrading from 1.x to 2.x may have issues.
2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all
the 2.x versions.
Actually I think there are some dependency version resolution policies out
there which picks the highest minor version when the dependencies pull in
multiple minor versions of the same jar, which may be broken if we remove
the API in minor releases.

I'm not sure about this. Yes, it's completely "legal" that we bump up the
> major version whenever a breaking change is needed. However, this also
> weakens the value of the commitment that public APIs will stay stable
> within the major release series, as the series can end anytime. IMHO, short
> major release series are not something "make the end users happy", but
> backdoors that allow us as the developers to make frequent breaking
> changes. On the contrary, with the demoting approach, we can still have
> longer major release series, while only allowing Public APIs deprecated at
> the end of the previous major version to be removed in the next major
> version.

I totally agree that frequent major version bumps are not ideal, but here
we are comparing it with a minor version bump which removes a Public API.
So the context is that we have already decided to remove this Public API
while keeping everything else backwards compatible. I think a major version
bump is a commonly understood signal here, compared with a minor version
change. From end users' perspective, for those who are not impacted, in
this case upgrading a major version is not necessarily more involved than
upgrading a minor version - both should be as smooth as a dependency
version change. For those who are impacted, they will lose the Public API
anyways and a major version bump ensures there is no surprise.

Thanks,

Jiangjie (Becket) Qin

On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com> wrote:

> Public API is a well defined common concept, and one of its
>> convention is that it only changes with a major version change.
>>
>
> I agree. And from my understanding, demoting a Public API is also a kind
> of such change, just like removing one, which can only happen with major
> version bumps. I'm not proposing to allow demoting Public APIs anytime, but
> only in the case major version bumps happen before reaching the
> 2-minor-release migration period. Actually, demoting would be a weaker
> change compared to removing the API immediately upon major version bumps,
> in order to keep the commitment about the 2-minor-release migration period.
> If the concern is that `@Public` -> `@PublicEvolving` sounds against
> conventions, we may introduce a new annotation if necessary, e.g.,
> `@PublicRetiring`, to avoid confusions.
>
> But it should be
>> completely OK to bump up the major version if we really want to get rid of
>> a public API, right?
>>
>
> I'm not sure about this. Yes, it's completely "legal" that we bump up the
> major version whenever a breaking change is needed. However, this also
> weakens the value of the commitment that public APIs will stay stable
> within the major release series, as the series can end anytime. IMHO, short
> major release series are not something "make the end users happy", but
> backdoors that allow us as the developers to make frequent breaking
> changes. On the contrary, with the demoting approach, we can still have
> longer major release series, while only allowing Public APIs deprecated at
> the end of the previous major version to be removed in the next major
> version.
>
> Given our track record I would prefer a regular cycle (1-2 years) to
>> force us to think about this whole topic, and not put it again to the
>> wayside and giving us (and users) a clear expectation on when breaking
>> changes can be made.
>>
>
> +1. I personally think 2-3 years would be a good time for new major
> versions, or longer if there's no breaking changes needed. That makes 1-2
> year a perfect time to revisit the topic, while leaving us more time to
> prepare the major release if needed.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> On 13/06/2023 17:26, Becket Qin wrote:
>> > It would be valuable if we can avoid releasing minor versions for
>> previous
>> > major versions.
>>
>> On paper, /absolutely /agree, but I'm not sure how viable that is in
>> practice.
>>
>> On the current 2.0 agenda is potentially dropping support for Java 8/11,
>> which may very well be a problem for our current users.
>>
>>
>> On 13/06/2023 17:26, Becket Qin wrote:
>> > Thanks for the feedback and sorry for the confusion about Public API
>> > deprecation. I just noticed that there was a mistake in the NOTES part
>> for
>> > Public API due to a copy-paste error... I just fixed it.
>> I'm very relieved to hear that. Glad to hear that we are on the same
>> page on that note.
>>
>>
>> On 15/06/2023 15:20, Becket Qin wrote:
>> > But it should be
>> > completely OK to bump up the major version if we really want to get rid
>> of
>> > a public API, right?
>>
>> Technically yes, but look at how long it took to get us to 2.0. ;)
>>
>> There's a separate discussion to be had on the cadence of major releases
>> going forward, and there seem to be different opinions on that.
>>
>> If we take the Kafka example of 2 minor releases between major ones,
>> that for us means that users have to potentially deal with breaking
>> changes every 6 months, which seems like a lot.
>>
>> Given our track record I would prefer a regular cycle (1-2 years) to
>> force us to think about this whole topic, and not put it again to the
>> wayside and giving us (and users) a clear expectation on when breaking
>> changes can be made.
>>
>> But again, maybe this should be in a separate thread.
>>
>> On 14/06/2023 11:37, Becket Qin wrote:
>> > Do you have an example of behavioral change in mind? Not sure I fully
>> > understand the concern for behavioral change here.
>>
>> This could be a lot of things. It can be performance in certain
>> edge-cases, a bug fix that users (maybe unknowingly) relied upon
>> (https://xkcd.com/1172/), a semantic change to some API.
>>
>> For a concrete example, consider the job submission. A few releases back
>> we made changes such that the initialization of the job master happens
>> asynchronously.
>> This meant the job submission call returns sooner, and the job state
>> enum was extended to cover this state.
>> API-wise we consider this a compatible change, but the observed behavior
>> may be different.
>>
>> Metrics are another example; I believe over time we changed what some
>> metrics returned a few times.
>>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Chesnay,

I think that there are two things we are discussing here:
1. The API stability story we WANT to have.
2. The API stability guarantees we CAN have.

We can only design for what we want. Good API stability with affordable
maintenance overhead does demand careful design from the high level
architecture to specific APIs. I also believe that the proposed stability
guarantees are achievable with good practices. I understand the
concern that there might be some existing code that makes it more difficult
for us to get where we want to be. In that case, I think we should discuss
how to improve the code, instead of compromising on what we want.

So I think it is valuable that we bring up the parts of the code that
blocks us and see if we can solve it.

On the current 2.0 agenda is potentially dropping support for Java 8/11,
> which may very well be a problem for our current users.

Java 8 support was deprecated in May 2022 with release 1.15. Assuming we
have 2.0 released by the end of 2024, there are 2.5 years of migration
window. If our survey shows that most people have migrated off from Java 8,
I think it is reasonable to drop 1.8 support. When doing so, that basically
means users on Java 8 will be stuck on 1.x, and not getting new features
because all the new features are going to be in the 2.x releases.
Personally I think this is reasonable, given the long migration window
there.

Technically yes, but look at how long it took to get us to 2.0. ;)
>
> There's a separate discussion to be had on the cadence of major releases
> going forward, and there seem to be different opinions on that.
>
> If we take the Kafka example of 2 minor releases between major ones, that
> for us means that users have to potentially deal with breaking changes
> every 6 months, which seems like a lot.
>
> Given our track record I would prefer a regular cycle (1-2 years) to force
> us to think about this whole topic, and not put it again to the wayside and
> giving us (and users) a clear expectation on when breaking changes can be
> made.
>
> But again, maybe this should be in a separate thread.
>
I agree it makes sense for us to review the necessity of a major release
with a regular cycle.


> For a concrete example, consider the job submission. A few releases back
> we made changes such that the initialization of the job master happens
> asynchronously.
> This meant the job submission call returns sooner, and the job state enum
> was extended to cover this state.
> API-wise we consider this a compatible change, but the observed behavior
> may be different.
> Metrics are another example; I believe over time we changed what some
> metrics returned a few times.


For the job submission example, here's what I think we should do:
1. If we consider dispatcher gateway as a public API as well, in the
DispatcherGateway, introduce a new RPC method version submitJobV2() for the
async submission. Otherwise, we can change the RPC method in place, maybe
with an option of async or not.
2. On the client side, have a separate method of submitAsync(), while
keeping the original synchronous API. Whether the sync API should be
removed or not is debatable as users may want to block on it to fail fast
in case of some failures. The implementation of submit() and submitAsync()
can potentially share most of the code.
3. Depending on how the JobStatus enum is exposed to the users, we may or
may not need to bump the API version of the related APIs as well. For
example, if we assume that users only query the job status after submit() /
submitAsync() returns, then we don't need to do anything because the
existing users only invoke  submit() which only returns after the job
status becomes CREATED. Therefore the new status of INITIALIZING is not
exposed to them. On the other hand, if we think users might query the job
status before submit() / submitAsync() returns, then we may need to create
RestfulApi.requestJobStatusV2() which may return Initializing status, while
we make sure RestfulApi.requestJobStatus() keeps the current behavior and
does not return Initializing status to the users.

Does this introduce maintenance overhead? Sure. But this is what Kafka has
been doing for the past 10 years. If you check the Kafka protocol guide[1],
it has all the versions of all the RPC requests/responses. Therefore the
client side behavior can be kept the same. Is it affordable? From my
experience, once you have this pattern setup, the maintenance overhead is
not that high.

Metrics are another example; I believe over time we changed what some
> metrics returned a few times.

For metrics usually it is easier. We just need to add a new metric
meanwhile deprecate the previous one.

Thanks again for raising these examples. This is a good discussion, as we
are getting to some root causes of our hesitation about the API stabilities.

Thanks,

Jiangjie (Becket) Qin


On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <to...@gmail.com> wrote:

> Public API is a well defined common concept, and one of its
>> convention is that it only changes with a major version change.
>>
>
> I agree. And from my understanding, demoting a Public API is also a kind
> of such change, just like removing one, which can only happen with major
> version bumps. I'm not proposing to allow demoting Public APIs anytime, but
> only in the case major version bumps happen before reaching the
> 2-minor-release migration period. Actually, demoting would be a weaker
> change compared to removing the API immediately upon major version bumps,
> in order to keep the commitment about the 2-minor-release migration period.
> If the concern is that `@Public` -> `@PublicEvolving` sounds against
> conventions, we may introduce a new annotation if necessary, e.g.,
> `@PublicRetiring`, to avoid confusions.
>
> But it should be
>> completely OK to bump up the major version if we really want to get rid of
>> a public API, right?
>>
>
> I'm not sure about this. Yes, it's completely "legal" that we bump up the
> major version whenever a breaking change is needed. However, this also
> weakens the value of the commitment that public APIs will stay stable
> within the major release series, as the series can end anytime. IMHO, short
> major release series are not something "make the end users happy", but
> backdoors that allow us as the developers to make frequent breaking
> changes. On the contrary, with the demoting approach, we can still have
> longer major release series, while only allowing Public APIs deprecated at
> the end of the previous major version to be removed in the next major
> version.
>
> Given our track record I would prefer a regular cycle (1-2 years) to
>> force us to think about this whole topic, and not put it again to the
>> wayside and giving us (and users) a clear expectation on when breaking
>> changes can be made.
>>
>
> +1. I personally think 2-3 years would be a good time for new major
> versions, or longer if there's no breaking changes needed. That makes 1-2
> year a perfect time to revisit the topic, while leaving us more time to
> prepare the major release if needed.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> On 13/06/2023 17:26, Becket Qin wrote:
>> > It would be valuable if we can avoid releasing minor versions for
>> previous
>> > major versions.
>>
>> On paper, /absolutely /agree, but I'm not sure how viable that is in
>> practice.
>>
>> On the current 2.0 agenda is potentially dropping support for Java 8/11,
>> which may very well be a problem for our current users.
>>
>>
>> On 13/06/2023 17:26, Becket Qin wrote:
>> > Thanks for the feedback and sorry for the confusion about Public API
>> > deprecation. I just noticed that there was a mistake in the NOTES part
>> for
>> > Public API due to a copy-paste error... I just fixed it.
>> I'm very relieved to hear that. Glad to hear that we are on the same
>> page on that note.
>>
>>
>> On 15/06/2023 15:20, Becket Qin wrote:
>> > But it should be
>> > completely OK to bump up the major version if we really want to get rid
>> of
>> > a public API, right?
>>
>> Technically yes, but look at how long it took to get us to 2.0. ;)
>>
>> There's a separate discussion to be had on the cadence of major releases
>> going forward, and there seem to be different opinions on that.
>>
>> If we take the Kafka example of 2 minor releases between major ones,
>> that for us means that users have to potentially deal with breaking
>> changes every 6 months, which seems like a lot.
>>
>> Given our track record I would prefer a regular cycle (1-2 years) to
>> force us to think about this whole topic, and not put it again to the
>> wayside and giving us (and users) a clear expectation on when breaking
>> changes can be made.
>>
>> But again, maybe this should be in a separate thread.
>>
>> On 14/06/2023 11:37, Becket Qin wrote:
>> > Do you have an example of behavioral change in mind? Not sure I fully
>> > understand the concern for behavioral change here.
>>
>> This could be a lot of things. It can be performance in certain
>> edge-cases, a bug fix that users (maybe unknowingly) relied upon
>> (https://xkcd.com/1172/), a semantic change to some API.
>>
>> For a concrete example, consider the job submission. A few releases back
>> we made changes such that the initialization of the job master happens
>> asynchronously.
>> This meant the job submission call returns sooner, and the job state
>> enum was extended to cover this state.
>> API-wise we consider this a compatible change, but the observed behavior
>> may be different.
>>
>> Metrics are another example; I believe over time we changed what some
>> metrics returned a few times.
>>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
>
> Public API is a well defined common concept, and one of its
> convention is that it only changes with a major version change.
>

I agree. And from my understanding, demoting a Public API is also a kind of
such change, just like removing one, which can only happen with major
version bumps. I'm not proposing to allow demoting Public APIs anytime, but
only in the case major version bumps happen before reaching the
2-minor-release migration period. Actually, demoting would be a weaker
change compared to removing the API immediately upon major version bumps,
in order to keep the commitment about the 2-minor-release migration period.
If the concern is that `@Public` -> `@PublicEvolving` sounds against
conventions, we may introduce a new annotation if necessary, e.g.,
`@PublicRetiring`, to avoid confusions.

But it should be
> completely OK to bump up the major version if we really want to get rid of
> a public API, right?
>

I'm not sure about this. Yes, it's completely "legal" that we bump up the
major version whenever a breaking change is needed. However, this also
weakens the value of the commitment that public APIs will stay stable
within the major release series, as the series can end anytime. IMHO, short
major release series are not something "make the end users happy", but
backdoors that allow us as the developers to make frequent breaking
changes. On the contrary, with the demoting approach, we can still have
longer major release series, while only allowing Public APIs deprecated at
the end of the previous major version to be removed in the next major
version.

Given our track record I would prefer a regular cycle (1-2 years) to
> force us to think about this whole topic, and not put it again to the
> wayside and giving us (and users) a clear expectation on when breaking
> changes can be made.
>

+1. I personally think 2-3 years would be a good time for new major
versions, or longer if there's no breaking changes needed. That makes 1-2
year a perfect time to revisit the topic, while leaving us more time to
prepare the major release if needed.

Best,

Xintong



On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler <ch...@apache.org>
wrote:

> On 13/06/2023 17:26, Becket Qin wrote:
> > It would be valuable if we can avoid releasing minor versions for
> previous
> > major versions.
>
> On paper, /absolutely /agree, but I'm not sure how viable that is in
> practice.
>
> On the current 2.0 agenda is potentially dropping support for Java 8/11,
> which may very well be a problem for our current users.
>
>
> On 13/06/2023 17:26, Becket Qin wrote:
> > Thanks for the feedback and sorry for the confusion about Public API
> > deprecation. I just noticed that there was a mistake in the NOTES part
> for
> > Public API due to a copy-paste error... I just fixed it.
> I'm very relieved to hear that. Glad to hear that we are on the same
> page on that note.
>
>
> On 15/06/2023 15:20, Becket Qin wrote:
> > But it should be
> > completely OK to bump up the major version if we really want to get rid
> of
> > a public API, right?
>
> Technically yes, but look at how long it took to get us to 2.0. ;)
>
> There's a separate discussion to be had on the cadence of major releases
> going forward, and there seem to be different opinions on that.
>
> If we take the Kafka example of 2 minor releases between major ones,
> that for us means that users have to potentially deal with breaking
> changes every 6 months, which seems like a lot.
>
> Given our track record I would prefer a regular cycle (1-2 years) to
> force us to think about this whole topic, and not put it again to the
> wayside and giving us (and users) a clear expectation on when breaking
> changes can be made.
>
> But again, maybe this should be in a separate thread.
>
> On 14/06/2023 11:37, Becket Qin wrote:
> > Do you have an example of behavioral change in mind? Not sure I fully
> > understand the concern for behavioral change here.
>
> This could be a lot of things. It can be performance in certain
> edge-cases, a bug fix that users (maybe unknowingly) relied upon
> (https://xkcd.com/1172/), a semantic change to some API.
>
> For a concrete example, consider the job submission. A few releases back
> we made changes such that the initialization of the job master happens
> asynchronously.
> This meant the job submission call returns sooner, and the job state
> enum was extended to cover this state.
> API-wise we consider this a compatible change, but the observed behavior
> may be different.
>
> Metrics are another example; I believe over time we changed what some
> metrics returned a few times.
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Chesnay Schepler <ch...@apache.org>.
On 13/06/2023 17:26, Becket Qin wrote:
> It would be valuable if we can avoid releasing minor versions for previous
> major versions.

On paper, /absolutely /agree, but I'm not sure how viable that is in 
practice.

On the current 2.0 agenda is potentially dropping support for Java 8/11, 
which may very well be a problem for our current users.


On 13/06/2023 17:26, Becket Qin wrote:
> Thanks for the feedback and sorry for the confusion about Public API
> deprecation. I just noticed that there was a mistake in the NOTES part for
> Public API due to a copy-paste error... I just fixed it.
I'm very relieved to hear that. Glad to hear that we are on the same 
page on that note.


On 15/06/2023 15:20, Becket Qin wrote:
> But it should be
> completely OK to bump up the major version if we really want to get rid of
> a public API, right?

Technically yes, but look at how long it took to get us to 2.0. ;)

There's a separate discussion to be had on the cadence of major releases 
going forward, and there seem to be different opinions on that.

If we take the Kafka example of 2 minor releases between major ones, 
that for us means that users have to potentially deal with breaking 
changes every 6 months, which seems like a lot.

Given our track record I would prefer a regular cycle (1-2 years) to 
force us to think about this whole topic, and not put it again to the 
wayside and giving us (and users) a clear expectation on when breaking 
changes can be made.

But again, maybe this should be in a separate thread.

On 14/06/2023 11:37, Becket Qin wrote:
> Do you have an example of behavioral change in mind? Not sure I fully
> understand the concern for behavioral change here.

This could be a lot of things. It can be performance in certain 
edge-cases, a bug fix that users (maybe unknowingly) relied upon 
(https://xkcd.com/1172/), a semantic change to some API.

For a concrete example, consider the job submission. A few releases back 
we made changes such that the initialization of the job master happens 
asynchronously.
This meant the job submission call returns sooner, and the job state 
enum was extended to cover this state.
API-wise we consider this a compatible change, but the observed behavior 
may be different.

Metrics are another example; I believe over time we changed what some 
metrics returned a few times.

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

I think the key of stability guarantees is about not breaking the
> commitments we made to users. From my understanding, when we mark an API as
> Public in for example 1.18, we make the following two commitments.
> a) The API will not be removed in all the 1.x serials.
> b) The API will be kept at least 2 minor releases after being marked
> Deprecated.

There is one more thing we need to follow which is not breaking the
conventions. Public API is a well defined common concept, and one of its
convention is that it only changes with a major version change. While it is
fine for us to define our own graduation path of an API like Experimental
=> PublicEvolving => Public, once an API reaches the Public API state, we
need to follow the common convention. Not every project has Experimental /
PublicEvolving APIs, even for those that do have similar annotations, I am
not aware of any demotion process. So the demotion or API stability is
something extra we put on our users, which is specific to Flink and breaks
the Public API convention.

As far as I understand, the reason for the demotion of a Public API is to
allow us to remove it without a major version bump. But it should be
completely OK to bump up the major version if we really want to get rid of
a public API, right? Isn't it just a version change which has almost no
additional cost to us? As an example, Kafka deprecated the old consumer in
0.11, and kept for 1.0 and 1.1, then the community bumped up the major
version to 2.0 and removed the code. So there are only two 1.x minor
versions, while 2.x has 8 minor versions and 3.x has 5 minor versions at
present.

I think the first priority about API stability is to make the end users
happy. Sometimes this does mean there will be maintenance overhead for us
as Flink maintainers. When there is a conflict and no way around, having
some trade-off is reasonable. However, in this particular case, there seems
no material benefit of having a stability demotion process while it does
weaken the user experience.

Thanks,

Jiangjie (Becket) Qin



On Thu, Jun 15, 2023 at 7:31 PM Martijn Visser <ma...@apache.org>
wrote:

> Hi all,
>
> First off, thanks for opening the discussion on this topic. I think it's an
> important one.
>
> From my perspective, I think that one of the major pain points that users
> have expressed over time, is the API stability of Flink overall. I think
> that every Flink user wants to have reliable, stable and good APIs, and
> that this should be an essential thing for the Flink community to achieve.
> I think there's still a lot of room to improve here. I think that Spark has
> a nice overview of considerations when to break an API [1]. What I really
> like and would emphasize is that they strive to avoid breaking APIs or
> silently changing behaviour. I think we should do the same for the Flink
> APIs.
>
> Overall, I would agree with the proposal regarding deprecation for both
> Experimental and PublicEvolving. I do think that we can only enforce this
> if we actually enforce the opposite as well (which is the promotion from
> Experimental -> PublicEvolving -> Public).
>
> What I don't think is a good principle, is the example where a Public API
> is deprecated in 1.20 and the next release is 2.0 with the requirement to
> keep it there until the next major release. I can see why it's being
> proposed (because it would avoid that the user needs to change their
> implementation towards a new API), the user is already faced with the
> situation that their implementation must be changed, given the fact that
> the community decided to go for a new major version. That already implies
> breaking changes and impact for the user. I think it's the primary reason
> why there is no Flink 2.0 yet.
>
> I'm not in favour of downgrading APIs from Public -> PublicEvolving ->
> Experimental. When doing that, we are breaking the contract with the Flink
> users who believe they are on an API that won't break, only to figure out a
> couple of releases later that this has actually happened.
>
> I believe we should treat API Stability as a first class citizen, so that
> each API is annotated (either Internal, else it's considered public with an
> annotation of either Experimental, PublicEvolving or Public) and users know
> how to rely on them. An accepted deprecation proposal will only help our
> users in understanding the guarantees that they can expect.
>
> Best regards,
>
> Martijn
>
> [1] https://spark.apache.org/versioning-policy.html
>
> On Thu, Jun 15, 2023 at 5:29 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > I agree that Public APIs should require a longer migration period. I
> think
> > that is why the FLIP requires at least 2 minor releases (compared to 1
> > minor release for PublicEvolving and 1 patch release for Experimental).
> >
> > I think the key of stability guarantees is about not breaking the
> > commitments we made to users. From my understanding, when we mark an API
> as
> > Public in for example 1.18, we make the following two commitments.
> > a) The API will not be removed in all the 1.x serials.
> > b) The API will be kept at least 2 minor releases after being marked
> > Deprecated.
> >
> > When there's a major release bump before 2 minor releases, a) is not
> > affected and, and we should only need to guarantee b) by keeping in the
> new
> > major release. This is the rationale behind my proposal.
> >
> > I think my proposal can provide the same migration experience for users
> as
> > if the major release bump has not happened. It should also not affect
> users
> > planning their usage of FlInk if we tell them at the beginning of the new
> > major release that the deprecated API will soon be removed.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Jun 14, 2023 at 10:01 PM Becket Qin <be...@gmail.com>
> wrote:
> >
> > > Thanks for the explanation, Matthias.
> > >
> > > In the example you raised, would it be better to just keep both YARN
> and
> > > K8S support in the new major version, but with YARN support deprecated
> if
> > > we want to? We can say for YARN we will only provide bug fixes but no
> > > feature development anymore. Given these two features are probably in
> two
> > > independent modules. Keeping both modules in the same new major version
> > > likely has zero additional cost compared with maintaining them in two
> > > different major versions respectively. This way we don't have the
> > > non-linear version issue, have fewer releases, and save a bunch of
> > > maintenance effort for multiple development branches.
> > >
> > > Regarding the stability demotion, I see your point. However, I still
> feel
> > > it a little weird that we demote a Public API to PublicEvolving just
> for
> > > the purpose of code removal in minor versions. This also results in
> some
> > > counter intuitive issues. For example, assuming users only use Public
> > APIs,
> > > they may be able to upgrade from 1.19.0 to 2.0 fine, but upgrading from
> > > 1.19 to 2.1 does not work because the Public API is removed, even
> though
> > > from the users' perspective, both of them are major version upgrades.
> So,
> > > in this case, I would rather bump up the major version again to remove
> > the
> > > deprecated Public API. That seems simpler and does not complicate the
> > well
> > > established versioning semantic conventions.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Jun 14, 2023 at 9:27 PM Matthias Pohl
> > > <ma...@aiven.io.invalid> wrote:
> > >
> > > > One (made-up) example from the top of my head would have been that
> the
> > > > community decides to focus fully on Kubernetes without considering
> Yarn
> > > > anymore because of some must-have feature on the Kubernetes side. At
> > the
> > > > same time there are still some users for whom it would be tricky to
> > > migrate
> > > > from Yarn to Kubernetes. Therefore, there would be some desire to
> still
> > > > maintain the older major version of Flink that supports Yarn.
> > > >
> > > > But now that I'm thinking about the example, I realize: A discussion
> on
> > > how
> > > > we would handle the two major versions in that case, could be started
> > > when
> > > > we actually run into this issue. It shouldn't be too hard to migrate
> > to a
> > > > non-linear versioning from what you are proposing if such a scenario
> > > comes
> > > > up.
> > > >
> > > > And on the topic of downgrading from @Public to @PublicEvolving:
> > > >
> > > > Demoting a Public API to PublicEvolving API sounds hacky. From what I
> > > > > understand the stability guarantee is not revocable because users
> > will
> > > > rely
> > > > > on the stability guarantee to plan their usage of Flink. Demoting
> an
> > > API
> > > > > essentially defeats the very purpose of stability guarantee to
> begin
> > > > with.
> > > > >
> > > >
> > > > I understand the stability guarantee in a way that it only applies
> > > within a
> > > > major version. Downgrading the stability constraint for an API with a
> > new
> > > > major version still seems to comply with the definition of a @Public
> > > > annotation as it's similar to changing the API in other ways. But I'm
> > not
> > > > insisting on that approach. I just found it a reasonable workaround.
> > > >
> > > > Thanks,
> > > > Matthias
> > > >
> > > > On Wed, Jun 14, 2023 at 11:38 AM Becket Qin <be...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Matthias,
> > > > >
> > > > > Thanks for the feedback.
> > > > >
> > > > > Do you have an example of behavioral change in mind? Not sure I
> fully
> > > > > understand the concern for behavioral change here. From what I
> > > > understand,
> > > > > any user sensible change in an existing API, regardless of its kind
> > > (API
> > > > > signature or behavioral change), can always be done in the
> following
> > > way:
> > > > >
> > > > > 1. Introduce a new API (new package, new class/interface, new
> method,
> > > new
> > > > > config, new metric, etc) while marking the old one as deprecated.
> > > > > 2. Let the new API and deprecated API coexist for the migration
> > period
> > > to
> > > > > allow planned migration from the users.
> > > > > 3. Remove the deprecated API.
> > > > >
> > > > > For example, Kafka deprecated its old consumer and replaced it
> with a
> > > new
> > > > > Consumer - basically everything changes. The source code of the old
> > > > > consumer was kept there for a few years across multiple major
> > versions.
> > > > > This does mean we have to keep both of the APIs for a few releases,
> > and
> > > > > even fix bugs in the old consumer, so additional maintenance effort
> > is
> > > > > required. But this allows the users to keep up with Kafka releases
> > > which
> > > > is
> > > > > extremely rewarding.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
> > > > > <ma...@aiven.io.invalid> wrote:
> > > > >
> > > > > > Thanks for starting this discussion, Becket. A few good points
> were
> > > > > raised.
> > > > > > Here's what I want to add:
> > > > > >
> > > > > > Stefan raised the point of behavioral stability (in contrast to
> API
> > > > > > stability). That might be a reason for users to not be able to go
> > > ahead
> > > > > > with a major version bump. Working around behavioral changes
> might
> > be
> > > > > > trickier than just switching from deprecated to newer APIs. I see
> > > your
> > > > > > motivation of having a more linear versioning even between major
> > > > versions
> > > > > > to avoid backports. Backports are painful enough for minor
> > versions.
> > > > > >
> > > > > > But with major versions becoming a thing in the Flink cosmos, I
> > could
> > > > > > imagine that the behavioral stability Stefan mentions actually
> > could
> > > > > become
> > > > > > a bigger issue: Major versions down the road might include bigger
> > > > > > behavioral changes which would prevent users from going ahead
> with
> > > the
> > > > > > major version bump. I understand that this is out of the original
> > > scope
> > > > > of
> > > > > > this FLIP. But nevertheless, it does support Chesnay's concerns
> > that
> > > a
> > > > > > linear versioning without maintaining older major versions might
> > not
> > > be
> > > > > > feasible. It sounds like we should have a discussion about how we
> > > treat
> > > > > > older major versions here (or have a dedicated discussion on that
> > > topic
> > > > > > before going ahead with that FLIP).
> > > > > >
> > > > > > On another note: I like Xintong's proposal of downgrading an API
> > > > > > from @Public to @PublicEvolving in the new major version. That
> > would
> > > > > allow
> > > > > > us to keep the original intention of the @Public annotation alive
> > > (i.e.
> > > > > > that those APIs are only removed in the next major version).
> > > > > >
> > > > > > Matthias
> > > > > >
> > > > > > On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <
> > tonysong820@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for bringing up this discussion, Becket.
> > > > > > >
> > > > > > > My two cents:
> > > > > > >
> > > > > > > 1. Do we allow deprecation & removal of APIs without adding a
> new
> > > one
> > > > > as
> > > > > > a
> > > > > > > replacement? The examples in the table give me an impression
> that
> > > > > marking
> > > > > > > an API as `@Deprecated` should only happen at the same time of
> > > > > > introducing
> > > > > > > a new replacing API, which I think is true in most but not all
> > the
> > > > > cases.
> > > > > > >
> > > > > > > If there is a major version bump before 2 minor releases in the
> > > > current
> > > > > > > > major version are reached, the major version should keep the
> > > source
> > > > > > code
> > > > > > > in
> > > > > > > > its own minor version until two minor versions are reached.
> For
> > > > > > example,
> > > > > > > in
> > > > > > > > the above case, if Flink 2.0 is released after 1.20, then the
> > > > > > deprecated
> > > > > > > > source code of foo will be kept in 2.0 and all the 2.x
> > versions.
> > > It
> > > > > can
> > > > > > > > only be removed in 3.0.
> > > > > > > >
> > > > > > >
> > > > > > > 2. I think this might be a bit too strict. For an API that we
> > > already
> > > > > > > decided to remove, having to keep it for all the 2.x versions
> > > simply
> > > > > > > because there's less than 2 minor releases between making the
> > > > decision
> > > > > > and
> > > > > > > the major release bump seems not necessarily. Alternatively,
> I'd
> > > like
> > > > > to
> > > > > > > propose to remove the `@Public` annotation (or downgrade it to
> > > > > > > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <
> becket.qin@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jing,
> > > > > > > >
> > > > > > > > Thanks for the feedback. Please see the answers to your
> > questions
> > > > > > below:
> > > > > > > >
> > > > > > > > *"Always add a "Since X.X.X" comment to indicate when was a
> > > class /
> > > > > > > > > interface / method marked as deprecated."*
> > > > > > > > >  Could you describe it with a code example? Do you mean
> Java
> > > > > > comments?
> > > > > > > >
> > > > > > > > It is just a comment such as "Since 1.18. Use X
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > > > > > > >XX
> > > > > > > > instead.". And we can then look it up in the deprecated
> list[1]
> > > in
> > > > > each
> > > > > > > > release and see which method should / can be deprecated.
> > > > > > > >
> > > > > > > > *"At least 1 patch release for the affected minor release for
> > > > > > > > > Experimental APIs"*
> > > > > > > > > The rule is absolutely right. However, afaiac, deprecation
> is
> > > > > > different
> > > > > > > > as
> > > > > > > > > modification. As a user/dev, I would appreciate, if I do
> not
> > > need
> > > > > to
> > > > > > do
> > > > > > > > any
> > > > > > > > > migration work for any deprecated API between patch
> releases
> > > > > upgrade.
> > > > > > > > BTW,
> > > > > > > > > if experimental APIs are allowed to change between patches,
> > > could
> > > > > we
> > > > > > > just
> > > > > > > > > change them instead of marking them as deprecated and
> create
> > > new
> > > > > ones
> > > > > > > to
> > > > > > > > > replace them?
> > > > > > > >
> > > > > > > > Deprecating an API is just a more elegant way of replacing an
> > API
> > > > > with
> > > > > > a
> > > > > > > > new one. The only difference between the two is whether the
> old
> > > API
> > > > > is
> > > > > > > kept
> > > > > > > > and coexists with the new API for some releases or not. For
> end
> > > > > users,
> > > > > > > > deprecation-then-remove is much more friendly than direct
> > > > > replacement.
> > > > > > > >
> > > > > > > > 1. How to make sure the new APIs cover all functionality,
> i.e.
> > > > > backward
> > > > > > > > > compatible, before removing the deprecated APIs? Since the
> > > > > > > > > functionalities could only be built with the new APIs
> > > > iteratively,
> > > > > > > there
> > > > > > > > > will be a while (might be longer than the migration period)
> > > that
> > > > > the
> > > > > > > new
> > > > > > > > > APIs are not backward compatible with the deprecated ones.
> > > > > > > >
> > > > > > > > This is orthogonal to the deprecation process, and may not
> even
> > > be
> > > > > > > required
> > > > > > > > in some cases if the function changes by itself. But in
> > general,
> > > > this
> > > > > > > > relies on the developer to decide. A simple test on readiness
> > is
> > > to
> > > > > see
> > > > > > > if
> > > > > > > > all the UT / IT cases written with the old API can be
> migrated
> > to
> > > > the
> > > > > > new
> > > > > > > > one and still work.  If the new API is not ready, we probably
> > > > should
> > > > > > not
> > > > > > > > deprecate the old one to begin with.
> > > > > > > >
> > > > > > > > 2. Is it allowed to remove the deprecated APIs after the
> > defined
> > > > > > > migration
> > > > > > > > > period expires while the new APis are still not backward
> > > > > compatible?
> > > > > > > >
> > > > > > > > By "backwards compatible", do you mean functionally
> equivalent?
> > > If
> > > > > the
> > > > > > > new
> > > > > > > > APIs are designed to be not backwards compatible, then
> removing
> > > the
> > > > > > > > deprecated source code is definitely allowed. If we don't
> think
> > > the
> > > > > new
> > > > > > > API
> > > > > > > > is ready to take over the place for the old one, then we
> should
> > > > wait.
> > > > > > The
> > > > > > > > migration period is the minimum time we have to wait before
> > > > removing
> > > > > > the
> > > > > > > > source code. A longer migration period is OK.
> > > > > > > >
> > > > > > > > 3. For the case of core API upgrade with downstream
> > > > implementations,
> > > > > > e.g.
> > > > > > > > > connectors, What is the feasible deprecation strategy?
> > Option1
> > > > > > > bottom-up:
> > > > > > > > > make sure the downstream implementation is backward
> > compatible
> > > > > before
> > > > > > > > > removing the deprecated core APIs. Option2 top-down: once
> the
> > > > > > > downstream
> > > > > > > > > implementation of new APIs works fine, we can remove the
> > > > deprecated
> > > > > > > core
> > > > > > > > > APIs after the migration period expires. The implementation
> > of
> > > > the
> > > > > > > > > deprecated APIs will not get any further update in upcoming
> > > > > > releases(it
> > > > > > > > has
> > > > > > > > > been removed). There might be some missing features in the
> > > > > downstream
> > > > > > > > > implementation of new APIs compared to the old
> > implementation.
> > > > Both
> > > > > > > > options
> > > > > > > > > have their own pros and cons.
> > > > > > > >
> > > > > > > > The downstream projects such as connectors in Flink should
> also
> > > > > follow
> > > > > > > the
> > > > > > > > migration path we tell our users. i.e. If there is a
> cascading
> > > > > > backwards
> > > > > > > > incompatible change in the connectors due to a backwards
> > > > incompatible
> > > > > > > > change in the core, and as a result a longer migration period
> > is
> > > > > > > required,
> > > > > > > > then I think we should postpone the removal of source code.
> But
> > > in
> > > > > > > general,
> > > > > > > > we should be able to provide the same migration period in the
> > > > > > connectors
> > > > > > > as
> > > > > > > > the flink core, if the connectors are upgraded to the latest
> > > > version
> > > > > of
> > > > > > > > core promptly.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > This is by design. Most of these are @Public APIs that we
> > had
> > > > to
> > > > > > > carry
> > > > > > > > > > around until Flink 2.0, because that was the initial
> > > guarantee
> > > > > that
> > > > > > > we
> > > > > > > > > > gave people.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > True, I knew @Public APIs could not be removed before the
> > next
> > > > > major
> > > > > > > > > release. I meant house cleaning without violation of these
> > > > > > annotations'
> > > > > > > > > design concept. i.e especially cleaning up for
> > @PublicEvolving
> > > > APIs
> > > > > > > since
> > > > > > > > > they are customer-facing. Regular cleaning up with all
> other
> > > > > > > > @Experimental
> > > > > > > > > and @Internal APIs would be even better, if there might be
> > some
> > > > > APIs
> > > > > > > > marked
> > > > > > > > > as @deprecated.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <
> > > > > chesnay@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > > > > > > One major issue we have, afaiu, is caused by the lack
> of
> > > > > > > > > > housekeeping/house
> > > > > > > > > > > cleaning, there are many APIs that were marked as
> > > deprecated
> > > > a
> > > > > > few
> > > > > > > > > years
> > > > > > > > > > > ago and still don't get removed. Some APIs should be
> easy
> > > to
> > > > > > remove
> > > > > > > > and
> > > > > > > > > > > others will need some more clear rules, like the issue
> > > > > discussed
> > > > > > at
> > > > > > > > > [1].
> > > > > > > > > >
> > > > > > > > > This is by design. Most of these are @Public APIs that we
> had
> > > to
> > > > > > carry
> > > > > > > > > > around until Flink 2.0, because that was the initial
> > > guarantee
> > > > > that
> > > > > > > we
> > > > > > > > > > gave people.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > As for the FLIP, I like the idea of explicitly writing
> > down a
> > > > > > > > > > deprecation period for APIs, particularly PublicEvolving
> > > ones.
> > > > > > > > > > For Experimental I don't think it'd be a problem if we
> > could
> > > > > change
> > > > > > > > them
> > > > > > > > > > right away,
> > > > > > > > > > but looking back a bit I don't think it hurts us to also
> > > > enforce
> > > > > > some
> > > > > > > > > > deprecation period.
> > > > > > > > > > 1 release for both of these sound fine to me.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > My major point of contention is the removal of Public
> APIs
> > > > > between
> > > > > > > > minor
> > > > > > > > > > versions.
> > > > > > > > > > This to me would a major setback towards a simpler
> upgrade
> > > path
> > > > > for
> > > > > > > > > users.
> > > > > > > > > > If these can be removed in minor versions than what even
> > is a
> > > > > major
> > > > > > > > > > release?
> > > > > > > > > > The very definition we have for Public APIs is that they
> > > stick
> > > > > > around
> > > > > > > > > > until the next major one.
> > > > > > > > > > Any rule that theoretically allows for breaking changes
> in
> > > > Public
> > > > > > API
> > > > > > > > in
> > > > > > > > > > every minor release is in my opinion not a viable option.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The "carry recent Public APIs forward into the next major
> > > > > release"
> > > > > > > > thing
> > > > > > > > > > seems to presume a linear release history (aka, if 2.0 is
> > > > > released
> > > > > > > > after
> > > > > > > > > > 1.20, then there will be no 1.21), which I doubt will be
> > the
> > > > > case.
> > > > > > > The
> > > > > > > > > > idea behind it is good, but I'd say the right conclusion
> > > would
> > > > be
> > > > > > to
> > > > > > > > not
> > > > > > > > > > make that API public if we know a new major release hits
> > in 3
> > > > > > months
> > > > > > > > and
> > > > > > > > > > is about to modify it. With a regular schedule for major
> > > > releases
> > > > > > > this
> > > > > > > > > > wouldn't be difficult to do.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Martijn Visser <ma...@apache.org>.
Hi all,

First off, thanks for opening the discussion on this topic. I think it's an
important one.

From my perspective, I think that one of the major pain points that users
have expressed over time, is the API stability of Flink overall. I think
that every Flink user wants to have reliable, stable and good APIs, and
that this should be an essential thing for the Flink community to achieve.
I think there's still a lot of room to improve here. I think that Spark has
a nice overview of considerations when to break an API [1]. What I really
like and would emphasize is that they strive to avoid breaking APIs or
silently changing behaviour. I think we should do the same for the Flink
APIs.

Overall, I would agree with the proposal regarding deprecation for both
Experimental and PublicEvolving. I do think that we can only enforce this
if we actually enforce the opposite as well (which is the promotion from
Experimental -> PublicEvolving -> Public).

What I don't think is a good principle, is the example where a Public API
is deprecated in 1.20 and the next release is 2.0 with the requirement to
keep it there until the next major release. I can see why it's being
proposed (because it would avoid that the user needs to change their
implementation towards a new API), the user is already faced with the
situation that their implementation must be changed, given the fact that
the community decided to go for a new major version. That already implies
breaking changes and impact for the user. I think it's the primary reason
why there is no Flink 2.0 yet.

I'm not in favour of downgrading APIs from Public -> PublicEvolving ->
Experimental. When doing that, we are breaking the contract with the Flink
users who believe they are on an API that won't break, only to figure out a
couple of releases later that this has actually happened.

I believe we should treat API Stability as a first class citizen, so that
each API is annotated (either Internal, else it's considered public with an
annotation of either Experimental, PublicEvolving or Public) and users know
how to rely on them. An accepted deprecation proposal will only help our
users in understanding the guarantees that they can expect.

Best regards,

Martijn

[1] https://spark.apache.org/versioning-policy.html

On Thu, Jun 15, 2023 at 5:29 AM Xintong Song <to...@gmail.com> wrote:

> I agree that Public APIs should require a longer migration period. I think
> that is why the FLIP requires at least 2 minor releases (compared to 1
> minor release for PublicEvolving and 1 patch release for Experimental).
>
> I think the key of stability guarantees is about not breaking the
> commitments we made to users. From my understanding, when we mark an API as
> Public in for example 1.18, we make the following two commitments.
> a) The API will not be removed in all the 1.x serials.
> b) The API will be kept at least 2 minor releases after being marked
> Deprecated.
>
> When there's a major release bump before 2 minor releases, a) is not
> affected and, and we should only need to guarantee b) by keeping in the new
> major release. This is the rationale behind my proposal.
>
> I think my proposal can provide the same migration experience for users as
> if the major release bump has not happened. It should also not affect users
> planning their usage of FlInk if we tell them at the beginning of the new
> major release that the deprecated API will soon be removed.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jun 14, 2023 at 10:01 PM Becket Qin <be...@gmail.com> wrote:
>
> > Thanks for the explanation, Matthias.
> >
> > In the example you raised, would it be better to just keep both YARN and
> > K8S support in the new major version, but with YARN support deprecated if
> > we want to? We can say for YARN we will only provide bug fixes but no
> > feature development anymore. Given these two features are probably in two
> > independent modules. Keeping both modules in the same new major version
> > likely has zero additional cost compared with maintaining them in two
> > different major versions respectively. This way we don't have the
> > non-linear version issue, have fewer releases, and save a bunch of
> > maintenance effort for multiple development branches.
> >
> > Regarding the stability demotion, I see your point. However, I still feel
> > it a little weird that we demote a Public API to PublicEvolving just for
> > the purpose of code removal in minor versions. This also results in some
> > counter intuitive issues. For example, assuming users only use Public
> APIs,
> > they may be able to upgrade from 1.19.0 to 2.0 fine, but upgrading from
> > 1.19 to 2.1 does not work because the Public API is removed, even though
> > from the users' perspective, both of them are major version upgrades. So,
> > in this case, I would rather bump up the major version again to remove
> the
> > deprecated Public API. That seems simpler and does not complicate the
> well
> > established versioning semantic conventions.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Jun 14, 2023 at 9:27 PM Matthias Pohl
> > <ma...@aiven.io.invalid> wrote:
> >
> > > One (made-up) example from the top of my head would have been that the
> > > community decides to focus fully on Kubernetes without considering Yarn
> > > anymore because of some must-have feature on the Kubernetes side. At
> the
> > > same time there are still some users for whom it would be tricky to
> > migrate
> > > from Yarn to Kubernetes. Therefore, there would be some desire to still
> > > maintain the older major version of Flink that supports Yarn.
> > >
> > > But now that I'm thinking about the example, I realize: A discussion on
> > how
> > > we would handle the two major versions in that case, could be started
> > when
> > > we actually run into this issue. It shouldn't be too hard to migrate
> to a
> > > non-linear versioning from what you are proposing if such a scenario
> > comes
> > > up.
> > >
> > > And on the topic of downgrading from @Public to @PublicEvolving:
> > >
> > > Demoting a Public API to PublicEvolving API sounds hacky. From what I
> > > > understand the stability guarantee is not revocable because users
> will
> > > rely
> > > > on the stability guarantee to plan their usage of Flink. Demoting an
> > API
> > > > essentially defeats the very purpose of stability guarantee to begin
> > > with.
> > > >
> > >
> > > I understand the stability guarantee in a way that it only applies
> > within a
> > > major version. Downgrading the stability constraint for an API with a
> new
> > > major version still seems to comply with the definition of a @Public
> > > annotation as it's similar to changing the API in other ways. But I'm
> not
> > > insisting on that approach. I just found it a reasonable workaround.
> > >
> > > Thanks,
> > > Matthias
> > >
> > > On Wed, Jun 14, 2023 at 11:38 AM Becket Qin <be...@gmail.com>
> > wrote:
> > >
> > > > Hi Matthias,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > Do you have an example of behavioral change in mind? Not sure I fully
> > > > understand the concern for behavioral change here. From what I
> > > understand,
> > > > any user sensible change in an existing API, regardless of its kind
> > (API
> > > > signature or behavioral change), can always be done in the following
> > way:
> > > >
> > > > 1. Introduce a new API (new package, new class/interface, new method,
> > new
> > > > config, new metric, etc) while marking the old one as deprecated.
> > > > 2. Let the new API and deprecated API coexist for the migration
> period
> > to
> > > > allow planned migration from the users.
> > > > 3. Remove the deprecated API.
> > > >
> > > > For example, Kafka deprecated its old consumer and replaced it with a
> > new
> > > > Consumer - basically everything changes. The source code of the old
> > > > consumer was kept there for a few years across multiple major
> versions.
> > > > This does mean we have to keep both of the APIs for a few releases,
> and
> > > > even fix bugs in the old consumer, so additional maintenance effort
> is
> > > > required. But this allows the users to keep up with Kafka releases
> > which
> > > is
> > > > extremely rewarding.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
> > > > <ma...@aiven.io.invalid> wrote:
> > > >
> > > > > Thanks for starting this discussion, Becket. A few good points were
> > > > raised.
> > > > > Here's what I want to add:
> > > > >
> > > > > Stefan raised the point of behavioral stability (in contrast to API
> > > > > stability). That might be a reason for users to not be able to go
> > ahead
> > > > > with a major version bump. Working around behavioral changes might
> be
> > > > > trickier than just switching from deprecated to newer APIs. I see
> > your
> > > > > motivation of having a more linear versioning even between major
> > > versions
> > > > > to avoid backports. Backports are painful enough for minor
> versions.
> > > > >
> > > > > But with major versions becoming a thing in the Flink cosmos, I
> could
> > > > > imagine that the behavioral stability Stefan mentions actually
> could
> > > > become
> > > > > a bigger issue: Major versions down the road might include bigger
> > > > > behavioral changes which would prevent users from going ahead with
> > the
> > > > > major version bump. I understand that this is out of the original
> > scope
> > > > of
> > > > > this FLIP. But nevertheless, it does support Chesnay's concerns
> that
> > a
> > > > > linear versioning without maintaining older major versions might
> not
> > be
> > > > > feasible. It sounds like we should have a discussion about how we
> > treat
> > > > > older major versions here (or have a dedicated discussion on that
> > topic
> > > > > before going ahead with that FLIP).
> > > > >
> > > > > On another note: I like Xintong's proposal of downgrading an API
> > > > > from @Public to @PublicEvolving in the new major version. That
> would
> > > > allow
> > > > > us to keep the original intention of the @Public annotation alive
> > (i.e.
> > > > > that those APIs are only removed in the next major version).
> > > > >
> > > > > Matthias
> > > > >
> > > > > On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <
> tonysong820@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for bringing up this discussion, Becket.
> > > > > >
> > > > > > My two cents:
> > > > > >
> > > > > > 1. Do we allow deprecation & removal of APIs without adding a new
> > one
> > > > as
> > > > > a
> > > > > > replacement? The examples in the table give me an impression that
> > > > marking
> > > > > > an API as `@Deprecated` should only happen at the same time of
> > > > > introducing
> > > > > > a new replacing API, which I think is true in most but not all
> the
> > > > cases.
> > > > > >
> > > > > > If there is a major version bump before 2 minor releases in the
> > > current
> > > > > > > major version are reached, the major version should keep the
> > source
> > > > > code
> > > > > > in
> > > > > > > its own minor version until two minor versions are reached. For
> > > > > example,
> > > > > > in
> > > > > > > the above case, if Flink 2.0 is released after 1.20, then the
> > > > > deprecated
> > > > > > > source code of foo will be kept in 2.0 and all the 2.x
> versions.
> > It
> > > > can
> > > > > > > only be removed in 3.0.
> > > > > > >
> > > > > >
> > > > > > 2. I think this might be a bit too strict. For an API that we
> > already
> > > > > > decided to remove, having to keep it for all the 2.x versions
> > simply
> > > > > > because there's less than 2 minor releases between making the
> > > decision
> > > > > and
> > > > > > the major release bump seems not necessarily. Alternatively, I'd
> > like
> > > > to
> > > > > > propose to remove the `@Public` annotation (or downgrade it to
> > > > > > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <becket.qin@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > > Thanks for the feedback. Please see the answers to your
> questions
> > > > > below:
> > > > > > >
> > > > > > > *"Always add a "Since X.X.X" comment to indicate when was a
> > class /
> > > > > > > > interface / method marked as deprecated."*
> > > > > > > >  Could you describe it with a code example? Do you mean Java
> > > > > comments?
> > > > > > >
> > > > > > > It is just a comment such as "Since 1.18. Use X
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > > > > > >XX
> > > > > > > instead.". And we can then look it up in the deprecated list[1]
> > in
> > > > each
> > > > > > > release and see which method should / can be deprecated.
> > > > > > >
> > > > > > > *"At least 1 patch release for the affected minor release for
> > > > > > > > Experimental APIs"*
> > > > > > > > The rule is absolutely right. However, afaiac, deprecation is
> > > > > different
> > > > > > > as
> > > > > > > > modification. As a user/dev, I would appreciate, if I do not
> > need
> > > > to
> > > > > do
> > > > > > > any
> > > > > > > > migration work for any deprecated API between patch releases
> > > > upgrade.
> > > > > > > BTW,
> > > > > > > > if experimental APIs are allowed to change between patches,
> > could
> > > > we
> > > > > > just
> > > > > > > > change them instead of marking them as deprecated and create
> > new
> > > > ones
> > > > > > to
> > > > > > > > replace them?
> > > > > > >
> > > > > > > Deprecating an API is just a more elegant way of replacing an
> API
> > > > with
> > > > > a
> > > > > > > new one. The only difference between the two is whether the old
> > API
> > > > is
> > > > > > kept
> > > > > > > and coexists with the new API for some releases or not. For end
> > > > users,
> > > > > > > deprecation-then-remove is much more friendly than direct
> > > > replacement.
> > > > > > >
> > > > > > > 1. How to make sure the new APIs cover all functionality, i.e.
> > > > backward
> > > > > > > > compatible, before removing the deprecated APIs? Since the
> > > > > > > > functionalities could only be built with the new APIs
> > > iteratively,
> > > > > > there
> > > > > > > > will be a while (might be longer than the migration period)
> > that
> > > > the
> > > > > > new
> > > > > > > > APIs are not backward compatible with the deprecated ones.
> > > > > > >
> > > > > > > This is orthogonal to the deprecation process, and may not even
> > be
> > > > > > required
> > > > > > > in some cases if the function changes by itself. But in
> general,
> > > this
> > > > > > > relies on the developer to decide. A simple test on readiness
> is
> > to
> > > > see
> > > > > > if
> > > > > > > all the UT / IT cases written with the old API can be migrated
> to
> > > the
> > > > > new
> > > > > > > one and still work.  If the new API is not ready, we probably
> > > should
> > > > > not
> > > > > > > deprecate the old one to begin with.
> > > > > > >
> > > > > > > 2. Is it allowed to remove the deprecated APIs after the
> defined
> > > > > > migration
> > > > > > > > period expires while the new APis are still not backward
> > > > compatible?
> > > > > > >
> > > > > > > By "backwards compatible", do you mean functionally equivalent?
> > If
> > > > the
> > > > > > new
> > > > > > > APIs are designed to be not backwards compatible, then removing
> > the
> > > > > > > deprecated source code is definitely allowed. If we don't think
> > the
> > > > new
> > > > > > API
> > > > > > > is ready to take over the place for the old one, then we should
> > > wait.
> > > > > The
> > > > > > > migration period is the minimum time we have to wait before
> > > removing
> > > > > the
> > > > > > > source code. A longer migration period is OK.
> > > > > > >
> > > > > > > 3. For the case of core API upgrade with downstream
> > > implementations,
> > > > > e.g.
> > > > > > > > connectors, What is the feasible deprecation strategy?
> Option1
> > > > > > bottom-up:
> > > > > > > > make sure the downstream implementation is backward
> compatible
> > > > before
> > > > > > > > removing the deprecated core APIs. Option2 top-down: once the
> > > > > > downstream
> > > > > > > > implementation of new APIs works fine, we can remove the
> > > deprecated
> > > > > > core
> > > > > > > > APIs after the migration period expires. The implementation
> of
> > > the
> > > > > > > > deprecated APIs will not get any further update in upcoming
> > > > > releases(it
> > > > > > > has
> > > > > > > > been removed). There might be some missing features in the
> > > > downstream
> > > > > > > > implementation of new APIs compared to the old
> implementation.
> > > Both
> > > > > > > options
> > > > > > > > have their own pros and cons.
> > > > > > >
> > > > > > > The downstream projects such as connectors in Flink should also
> > > > follow
> > > > > > the
> > > > > > > migration path we tell our users. i.e. If there is a cascading
> > > > > backwards
> > > > > > > incompatible change in the connectors due to a backwards
> > > incompatible
> > > > > > > change in the core, and as a result a longer migration period
> is
> > > > > > required,
> > > > > > > then I think we should postpone the removal of source code. But
> > in
> > > > > > general,
> > > > > > > we should be able to provide the same migration period in the
> > > > > connectors
> > > > > > as
> > > > > > > the flink core, if the connectors are upgraded to the latest
> > > version
> > > > of
> > > > > > > core promptly.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > This is by design. Most of these are @Public APIs that we
> had
> > > to
> > > > > > carry
> > > > > > > > > around until Flink 2.0, because that was the initial
> > guarantee
> > > > that
> > > > > > we
> > > > > > > > > gave people.
> > > > > > > > >
> > > > > > > >
> > > > > > > > True, I knew @Public APIs could not be removed before the
> next
> > > > major
> > > > > > > > release. I meant house cleaning without violation of these
> > > > > annotations'
> > > > > > > > design concept. i.e especially cleaning up for
> @PublicEvolving
> > > APIs
> > > > > > since
> > > > > > > > they are customer-facing. Regular cleaning up with all other
> > > > > > > @Experimental
> > > > > > > > and @Internal APIs would be even better, if there might be
> some
> > > > APIs
> > > > > > > marked
> > > > > > > > as @deprecated.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <
> > > > chesnay@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > > > > > One major issue we have, afaiu, is caused by the lack of
> > > > > > > > > housekeeping/house
> > > > > > > > > > cleaning, there are many APIs that were marked as
> > deprecated
> > > a
> > > > > few
> > > > > > > > years
> > > > > > > > > > ago and still don't get removed. Some APIs should be easy
> > to
> > > > > remove
> > > > > > > and
> > > > > > > > > > others will need some more clear rules, like the issue
> > > > discussed
> > > > > at
> > > > > > > > [1].
> > > > > > > > >
> > > > > > > > This is by design. Most of these are @Public APIs that we had
> > to
> > > > > carry
> > > > > > > > > around until Flink 2.0, because that was the initial
> > guarantee
> > > > that
> > > > > > we
> > > > > > > > > gave people.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > As for the FLIP, I like the idea of explicitly writing
> down a
> > > > > > > > > deprecation period for APIs, particularly PublicEvolving
> > ones.
> > > > > > > > > For Experimental I don't think it'd be a problem if we
> could
> > > > change
> > > > > > > them
> > > > > > > > > right away,
> > > > > > > > > but looking back a bit I don't think it hurts us to also
> > > enforce
> > > > > some
> > > > > > > > > deprecation period.
> > > > > > > > > 1 release for both of these sound fine to me.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > My major point of contention is the removal of Public APIs
> > > > between
> > > > > > > minor
> > > > > > > > > versions.
> > > > > > > > > This to me would a major setback towards a simpler upgrade
> > path
> > > > for
> > > > > > > > users.
> > > > > > > > > If these can be removed in minor versions than what even
> is a
> > > > major
> > > > > > > > > release?
> > > > > > > > > The very definition we have for Public APIs is that they
> > stick
> > > > > around
> > > > > > > > > until the next major one.
> > > > > > > > > Any rule that theoretically allows for breaking changes in
> > > Public
> > > > > API
> > > > > > > in
> > > > > > > > > every minor release is in my opinion not a viable option.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The "carry recent Public APIs forward into the next major
> > > > release"
> > > > > > > thing
> > > > > > > > > seems to presume a linear release history (aka, if 2.0 is
> > > > released
> > > > > > > after
> > > > > > > > > 1.20, then there will be no 1.21), which I doubt will be
> the
> > > > case.
> > > > > > The
> > > > > > > > > idea behind it is good, but I'd say the right conclusion
> > would
> > > be
> > > > > to
> > > > > > > not
> > > > > > > > > make that API public if we know a new major release hits
> in 3
> > > > > months
> > > > > > > and
> > > > > > > > > is about to modify it. With a regular schedule for major
> > > releases
> > > > > > this
> > > > > > > > > wouldn't be difficult to do.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
I agree that Public APIs should require a longer migration period. I think
that is why the FLIP requires at least 2 minor releases (compared to 1
minor release for PublicEvolving and 1 patch release for Experimental).

I think the key of stability guarantees is about not breaking the
commitments we made to users. From my understanding, when we mark an API as
Public in for example 1.18, we make the following two commitments.
a) The API will not be removed in all the 1.x serials.
b) The API will be kept at least 2 minor releases after being marked
Deprecated.

When there's a major release bump before 2 minor releases, a) is not
affected and, and we should only need to guarantee b) by keeping in the new
major release. This is the rationale behind my proposal.

I think my proposal can provide the same migration experience for users as
if the major release bump has not happened. It should also not affect users
planning their usage of FlInk if we tell them at the beginning of the new
major release that the deprecated API will soon be removed.

Best,

Xintong



On Wed, Jun 14, 2023 at 10:01 PM Becket Qin <be...@gmail.com> wrote:

> Thanks for the explanation, Matthias.
>
> In the example you raised, would it be better to just keep both YARN and
> K8S support in the new major version, but with YARN support deprecated if
> we want to? We can say for YARN we will only provide bug fixes but no
> feature development anymore. Given these two features are probably in two
> independent modules. Keeping both modules in the same new major version
> likely has zero additional cost compared with maintaining them in two
> different major versions respectively. This way we don't have the
> non-linear version issue, have fewer releases, and save a bunch of
> maintenance effort for multiple development branches.
>
> Regarding the stability demotion, I see your point. However, I still feel
> it a little weird that we demote a Public API to PublicEvolving just for
> the purpose of code removal in minor versions. This also results in some
> counter intuitive issues. For example, assuming users only use Public APIs,
> they may be able to upgrade from 1.19.0 to 2.0 fine, but upgrading from
> 1.19 to 2.1 does not work because the Public API is removed, even though
> from the users' perspective, both of them are major version upgrades. So,
> in this case, I would rather bump up the major version again to remove the
> deprecated Public API. That seems simpler and does not complicate the well
> established versioning semantic conventions.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jun 14, 2023 at 9:27 PM Matthias Pohl
> <ma...@aiven.io.invalid> wrote:
>
> > One (made-up) example from the top of my head would have been that the
> > community decides to focus fully on Kubernetes without considering Yarn
> > anymore because of some must-have feature on the Kubernetes side. At the
> > same time there are still some users for whom it would be tricky to
> migrate
> > from Yarn to Kubernetes. Therefore, there would be some desire to still
> > maintain the older major version of Flink that supports Yarn.
> >
> > But now that I'm thinking about the example, I realize: A discussion on
> how
> > we would handle the two major versions in that case, could be started
> when
> > we actually run into this issue. It shouldn't be too hard to migrate to a
> > non-linear versioning from what you are proposing if such a scenario
> comes
> > up.
> >
> > And on the topic of downgrading from @Public to @PublicEvolving:
> >
> > Demoting a Public API to PublicEvolving API sounds hacky. From what I
> > > understand the stability guarantee is not revocable because users will
> > rely
> > > on the stability guarantee to plan their usage of Flink. Demoting an
> API
> > > essentially defeats the very purpose of stability guarantee to begin
> > with.
> > >
> >
> > I understand the stability guarantee in a way that it only applies
> within a
> > major version. Downgrading the stability constraint for an API with a new
> > major version still seems to comply with the definition of a @Public
> > annotation as it's similar to changing the API in other ways. But I'm not
> > insisting on that approach. I just found it a reasonable workaround.
> >
> > Thanks,
> > Matthias
> >
> > On Wed, Jun 14, 2023 at 11:38 AM Becket Qin <be...@gmail.com>
> wrote:
> >
> > > Hi Matthias,
> > >
> > > Thanks for the feedback.
> > >
> > > Do you have an example of behavioral change in mind? Not sure I fully
> > > understand the concern for behavioral change here. From what I
> > understand,
> > > any user sensible change in an existing API, regardless of its kind
> (API
> > > signature or behavioral change), can always be done in the following
> way:
> > >
> > > 1. Introduce a new API (new package, new class/interface, new method,
> new
> > > config, new metric, etc) while marking the old one as deprecated.
> > > 2. Let the new API and deprecated API coexist for the migration period
> to
> > > allow planned migration from the users.
> > > 3. Remove the deprecated API.
> > >
> > > For example, Kafka deprecated its old consumer and replaced it with a
> new
> > > Consumer - basically everything changes. The source code of the old
> > > consumer was kept there for a few years across multiple major versions.
> > > This does mean we have to keep both of the APIs for a few releases, and
> > > even fix bugs in the old consumer, so additional maintenance effort is
> > > required. But this allows the users to keep up with Kafka releases
> which
> > is
> > > extremely rewarding.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
> > > <ma...@aiven.io.invalid> wrote:
> > >
> > > > Thanks for starting this discussion, Becket. A few good points were
> > > raised.
> > > > Here's what I want to add:
> > > >
> > > > Stefan raised the point of behavioral stability (in contrast to API
> > > > stability). That might be a reason for users to not be able to go
> ahead
> > > > with a major version bump. Working around behavioral changes might be
> > > > trickier than just switching from deprecated to newer APIs. I see
> your
> > > > motivation of having a more linear versioning even between major
> > versions
> > > > to avoid backports. Backports are painful enough for minor versions.
> > > >
> > > > But with major versions becoming a thing in the Flink cosmos, I could
> > > > imagine that the behavioral stability Stefan mentions actually could
> > > become
> > > > a bigger issue: Major versions down the road might include bigger
> > > > behavioral changes which would prevent users from going ahead with
> the
> > > > major version bump. I understand that this is out of the original
> scope
> > > of
> > > > this FLIP. But nevertheless, it does support Chesnay's concerns that
> a
> > > > linear versioning without maintaining older major versions might not
> be
> > > > feasible. It sounds like we should have a discussion about how we
> treat
> > > > older major versions here (or have a dedicated discussion on that
> topic
> > > > before going ahead with that FLIP).
> > > >
> > > > On another note: I like Xintong's proposal of downgrading an API
> > > > from @Public to @PublicEvolving in the new major version. That would
> > > allow
> > > > us to keep the original intention of the @Public annotation alive
> (i.e.
> > > > that those APIs are only removed in the next major version).
> > > >
> > > > Matthias
> > > >
> > > > On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <tonysong820@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks for bringing up this discussion, Becket.
> > > > >
> > > > > My two cents:
> > > > >
> > > > > 1. Do we allow deprecation & removal of APIs without adding a new
> one
> > > as
> > > > a
> > > > > replacement? The examples in the table give me an impression that
> > > marking
> > > > > an API as `@Deprecated` should only happen at the same time of
> > > > introducing
> > > > > a new replacing API, which I think is true in most but not all the
> > > cases.
> > > > >
> > > > > If there is a major version bump before 2 minor releases in the
> > current
> > > > > > major version are reached, the major version should keep the
> source
> > > > code
> > > > > in
> > > > > > its own minor version until two minor versions are reached. For
> > > > example,
> > > > > in
> > > > > > the above case, if Flink 2.0 is released after 1.20, then the
> > > > deprecated
> > > > > > source code of foo will be kept in 2.0 and all the 2.x versions.
> It
> > > can
> > > > > > only be removed in 3.0.
> > > > > >
> > > > >
> > > > > 2. I think this might be a bit too strict. For an API that we
> already
> > > > > decided to remove, having to keep it for all the 2.x versions
> simply
> > > > > because there's less than 2 minor releases between making the
> > decision
> > > > and
> > > > > the major release bump seems not necessarily. Alternatively, I'd
> like
> > > to
> > > > > propose to remove the `@Public` annotation (or downgrade it to
> > > > > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > > Thanks for the feedback. Please see the answers to your questions
> > > > below:
> > > > > >
> > > > > > *"Always add a "Since X.X.X" comment to indicate when was a
> class /
> > > > > > > interface / method marked as deprecated."*
> > > > > > >  Could you describe it with a code example? Do you mean Java
> > > > comments?
> > > > > >
> > > > > > It is just a comment such as "Since 1.18. Use X
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > > > > >XX
> > > > > > instead.". And we can then look it up in the deprecated list[1]
> in
> > > each
> > > > > > release and see which method should / can be deprecated.
> > > > > >
> > > > > > *"At least 1 patch release for the affected minor release for
> > > > > > > Experimental APIs"*
> > > > > > > The rule is absolutely right. However, afaiac, deprecation is
> > > > different
> > > > > > as
> > > > > > > modification. As a user/dev, I would appreciate, if I do not
> need
> > > to
> > > > do
> > > > > > any
> > > > > > > migration work for any deprecated API between patch releases
> > > upgrade.
> > > > > > BTW,
> > > > > > > if experimental APIs are allowed to change between patches,
> could
> > > we
> > > > > just
> > > > > > > change them instead of marking them as deprecated and create
> new
> > > ones
> > > > > to
> > > > > > > replace them?
> > > > > >
> > > > > > Deprecating an API is just a more elegant way of replacing an API
> > > with
> > > > a
> > > > > > new one. The only difference between the two is whether the old
> API
> > > is
> > > > > kept
> > > > > > and coexists with the new API for some releases or not. For end
> > > users,
> > > > > > deprecation-then-remove is much more friendly than direct
> > > replacement.
> > > > > >
> > > > > > 1. How to make sure the new APIs cover all functionality, i.e.
> > > backward
> > > > > > > compatible, before removing the deprecated APIs? Since the
> > > > > > > functionalities could only be built with the new APIs
> > iteratively,
> > > > > there
> > > > > > > will be a while (might be longer than the migration period)
> that
> > > the
> > > > > new
> > > > > > > APIs are not backward compatible with the deprecated ones.
> > > > > >
> > > > > > This is orthogonal to the deprecation process, and may not even
> be
> > > > > required
> > > > > > in some cases if the function changes by itself. But in general,
> > this
> > > > > > relies on the developer to decide. A simple test on readiness is
> to
> > > see
> > > > > if
> > > > > > all the UT / IT cases written with the old API can be migrated to
> > the
> > > > new
> > > > > > one and still work.  If the new API is not ready, we probably
> > should
> > > > not
> > > > > > deprecate the old one to begin with.
> > > > > >
> > > > > > 2. Is it allowed to remove the deprecated APIs after the defined
> > > > > migration
> > > > > > > period expires while the new APis are still not backward
> > > compatible?
> > > > > >
> > > > > > By "backwards compatible", do you mean functionally equivalent?
> If
> > > the
> > > > > new
> > > > > > APIs are designed to be not backwards compatible, then removing
> the
> > > > > > deprecated source code is definitely allowed. If we don't think
> the
> > > new
> > > > > API
> > > > > > is ready to take over the place for the old one, then we should
> > wait.
> > > > The
> > > > > > migration period is the minimum time we have to wait before
> > removing
> > > > the
> > > > > > source code. A longer migration period is OK.
> > > > > >
> > > > > > 3. For the case of core API upgrade with downstream
> > implementations,
> > > > e.g.
> > > > > > > connectors, What is the feasible deprecation strategy? Option1
> > > > > bottom-up:
> > > > > > > make sure the downstream implementation is backward compatible
> > > before
> > > > > > > removing the deprecated core APIs. Option2 top-down: once the
> > > > > downstream
> > > > > > > implementation of new APIs works fine, we can remove the
> > deprecated
> > > > > core
> > > > > > > APIs after the migration period expires. The implementation of
> > the
> > > > > > > deprecated APIs will not get any further update in upcoming
> > > > releases(it
> > > > > > has
> > > > > > > been removed). There might be some missing features in the
> > > downstream
> > > > > > > implementation of new APIs compared to the old implementation.
> > Both
> > > > > > options
> > > > > > > have their own pros and cons.
> > > > > >
> > > > > > The downstream projects such as connectors in Flink should also
> > > follow
> > > > > the
> > > > > > migration path we tell our users. i.e. If there is a cascading
> > > > backwards
> > > > > > incompatible change in the connectors due to a backwards
> > incompatible
> > > > > > change in the core, and as a result a longer migration period is
> > > > > required,
> > > > > > then I think we should postpone the removal of source code. But
> in
> > > > > general,
> > > > > > we should be able to provide the same migration period in the
> > > > connectors
> > > > > as
> > > > > > the flink core, if the connectors are upgraded to the latest
> > version
> > > of
> > > > > > core promptly.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > > > > >
> > > > > >
> > > > > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > > This is by design. Most of these are @Public APIs that we had
> > to
> > > > > carry
> > > > > > > > around until Flink 2.0, because that was the initial
> guarantee
> > > that
> > > > > we
> > > > > > > > gave people.
> > > > > > > >
> > > > > > >
> > > > > > > True, I knew @Public APIs could not be removed before the next
> > > major
> > > > > > > release. I meant house cleaning without violation of these
> > > > annotations'
> > > > > > > design concept. i.e especially cleaning up for @PublicEvolving
> > APIs
> > > > > since
> > > > > > > they are customer-facing. Regular cleaning up with all other
> > > > > > @Experimental
> > > > > > > and @Internal APIs would be even better, if there might be some
> > > APIs
> > > > > > marked
> > > > > > > as @deprecated.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <
> > > chesnay@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > > > > One major issue we have, afaiu, is caused by the lack of
> > > > > > > > housekeeping/house
> > > > > > > > > cleaning, there are many APIs that were marked as
> deprecated
> > a
> > > > few
> > > > > > > years
> > > > > > > > > ago and still don't get removed. Some APIs should be easy
> to
> > > > remove
> > > > > > and
> > > > > > > > > others will need some more clear rules, like the issue
> > > discussed
> > > > at
> > > > > > > [1].
> > > > > > > >
> > > > > > > This is by design. Most of these are @Public APIs that we had
> to
> > > > carry
> > > > > > > > around until Flink 2.0, because that was the initial
> guarantee
> > > that
> > > > > we
> > > > > > > > gave people.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > > > > > deprecation period for APIs, particularly PublicEvolving
> ones.
> > > > > > > > For Experimental I don't think it'd be a problem if we could
> > > change
> > > > > > them
> > > > > > > > right away,
> > > > > > > > but looking back a bit I don't think it hurts us to also
> > enforce
> > > > some
> > > > > > > > deprecation period.
> > > > > > > > 1 release for both of these sound fine to me.
> > > > > > > >
> > > > > > > >
> > > > > > > > My major point of contention is the removal of Public APIs
> > > between
> > > > > > minor
> > > > > > > > versions.
> > > > > > > > This to me would a major setback towards a simpler upgrade
> path
> > > for
> > > > > > > users.
> > > > > > > > If these can be removed in minor versions than what even is a
> > > major
> > > > > > > > release?
> > > > > > > > The very definition we have for Public APIs is that they
> stick
> > > > around
> > > > > > > > until the next major one.
> > > > > > > > Any rule that theoretically allows for breaking changes in
> > Public
> > > > API
> > > > > > in
> > > > > > > > every minor release is in my opinion not a viable option.
> > > > > > > >
> > > > > > > >
> > > > > > > > The "carry recent Public APIs forward into the next major
> > > release"
> > > > > > thing
> > > > > > > > seems to presume a linear release history (aka, if 2.0 is
> > > released
> > > > > > after
> > > > > > > > 1.20, then there will be no 1.21), which I doubt will be the
> > > case.
> > > > > The
> > > > > > > > idea behind it is good, but I'd say the right conclusion
> would
> > be
> > > > to
> > > > > > not
> > > > > > > > make that API public if we know a new major release hits in 3
> > > > months
> > > > > > and
> > > > > > > > is about to modify it. With a regular schedule for major
> > releases
> > > > > this
> > > > > > > > wouldn't be difficult to do.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Thanks for the explanation, Matthias.

In the example you raised, would it be better to just keep both YARN and
K8S support in the new major version, but with YARN support deprecated if
we want to? We can say for YARN we will only provide bug fixes but no
feature development anymore. Given these two features are probably in two
independent modules. Keeping both modules in the same new major version
likely has zero additional cost compared with maintaining them in two
different major versions respectively. This way we don't have the
non-linear version issue, have fewer releases, and save a bunch of
maintenance effort for multiple development branches.

Regarding the stability demotion, I see your point. However, I still feel
it a little weird that we demote a Public API to PublicEvolving just for
the purpose of code removal in minor versions. This also results in some
counter intuitive issues. For example, assuming users only use Public APIs,
they may be able to upgrade from 1.19.0 to 2.0 fine, but upgrading from
1.19 to 2.1 does not work because the Public API is removed, even though
from the users' perspective, both of them are major version upgrades. So,
in this case, I would rather bump up the major version again to remove the
deprecated Public API. That seems simpler and does not complicate the well
established versioning semantic conventions.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jun 14, 2023 at 9:27 PM Matthias Pohl
<ma...@aiven.io.invalid> wrote:

> One (made-up) example from the top of my head would have been that the
> community decides to focus fully on Kubernetes without considering Yarn
> anymore because of some must-have feature on the Kubernetes side. At the
> same time there are still some users for whom it would be tricky to migrate
> from Yarn to Kubernetes. Therefore, there would be some desire to still
> maintain the older major version of Flink that supports Yarn.
>
> But now that I'm thinking about the example, I realize: A discussion on how
> we would handle the two major versions in that case, could be started when
> we actually run into this issue. It shouldn't be too hard to migrate to a
> non-linear versioning from what you are proposing if such a scenario comes
> up.
>
> And on the topic of downgrading from @Public to @PublicEvolving:
>
> Demoting a Public API to PublicEvolving API sounds hacky. From what I
> > understand the stability guarantee is not revocable because users will
> rely
> > on the stability guarantee to plan their usage of Flink. Demoting an API
> > essentially defeats the very purpose of stability guarantee to begin
> with.
> >
>
> I understand the stability guarantee in a way that it only applies within a
> major version. Downgrading the stability constraint for an API with a new
> major version still seems to comply with the definition of a @Public
> annotation as it's similar to changing the API in other ways. But I'm not
> insisting on that approach. I just found it a reasonable workaround.
>
> Thanks,
> Matthias
>
> On Wed, Jun 14, 2023 at 11:38 AM Becket Qin <be...@gmail.com> wrote:
>
> > Hi Matthias,
> >
> > Thanks for the feedback.
> >
> > Do you have an example of behavioral change in mind? Not sure I fully
> > understand the concern for behavioral change here. From what I
> understand,
> > any user sensible change in an existing API, regardless of its kind (API
> > signature or behavioral change), can always be done in the following way:
> >
> > 1. Introduce a new API (new package, new class/interface, new method, new
> > config, new metric, etc) while marking the old one as deprecated.
> > 2. Let the new API and deprecated API coexist for the migration period to
> > allow planned migration from the users.
> > 3. Remove the deprecated API.
> >
> > For example, Kafka deprecated its old consumer and replaced it with a new
> > Consumer - basically everything changes. The source code of the old
> > consumer was kept there for a few years across multiple major versions.
> > This does mean we have to keep both of the APIs for a few releases, and
> > even fix bugs in the old consumer, so additional maintenance effort is
> > required. But this allows the users to keep up with Kafka releases which
> is
> > extremely rewarding.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
> > <ma...@aiven.io.invalid> wrote:
> >
> > > Thanks for starting this discussion, Becket. A few good points were
> > raised.
> > > Here's what I want to add:
> > >
> > > Stefan raised the point of behavioral stability (in contrast to API
> > > stability). That might be a reason for users to not be able to go ahead
> > > with a major version bump. Working around behavioral changes might be
> > > trickier than just switching from deprecated to newer APIs. I see your
> > > motivation of having a more linear versioning even between major
> versions
> > > to avoid backports. Backports are painful enough for minor versions.
> > >
> > > But with major versions becoming a thing in the Flink cosmos, I could
> > > imagine that the behavioral stability Stefan mentions actually could
> > become
> > > a bigger issue: Major versions down the road might include bigger
> > > behavioral changes which would prevent users from going ahead with the
> > > major version bump. I understand that this is out of the original scope
> > of
> > > this FLIP. But nevertheless, it does support Chesnay's concerns that a
> > > linear versioning without maintaining older major versions might not be
> > > feasible. It sounds like we should have a discussion about how we treat
> > > older major versions here (or have a dedicated discussion on that topic
> > > before going ahead with that FLIP).
> > >
> > > On another note: I like Xintong's proposal of downgrading an API
> > > from @Public to @PublicEvolving in the new major version. That would
> > allow
> > > us to keep the original intention of the @Public annotation alive (i.e.
> > > that those APIs are only removed in the next major version).
> > >
> > > Matthias
> > >
> > > On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <to...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for bringing up this discussion, Becket.
> > > >
> > > > My two cents:
> > > >
> > > > 1. Do we allow deprecation & removal of APIs without adding a new one
> > as
> > > a
> > > > replacement? The examples in the table give me an impression that
> > marking
> > > > an API as `@Deprecated` should only happen at the same time of
> > > introducing
> > > > a new replacing API, which I think is true in most but not all the
> > cases.
> > > >
> > > > If there is a major version bump before 2 minor releases in the
> current
> > > > > major version are reached, the major version should keep the source
> > > code
> > > > in
> > > > > its own minor version until two minor versions are reached. For
> > > example,
> > > > in
> > > > > the above case, if Flink 2.0 is released after 1.20, then the
> > > deprecated
> > > > > source code of foo will be kept in 2.0 and all the 2.x versions. It
> > can
> > > > > only be removed in 3.0.
> > > > >
> > > >
> > > > 2. I think this might be a bit too strict. For an API that we already
> > > > decided to remove, having to keep it for all the 2.x versions simply
> > > > because there's less than 2 minor releases between making the
> decision
> > > and
> > > > the major release bump seems not necessarily. Alternatively, I'd like
> > to
> > > > propose to remove the `@Public` annotation (or downgrade it to
> > > > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > Thanks for the feedback. Please see the answers to your questions
> > > below:
> > > > >
> > > > > *"Always add a "Since X.X.X" comment to indicate when was a class /
> > > > > > interface / method marked as deprecated."*
> > > > > >  Could you describe it with a code example? Do you mean Java
> > > comments?
> > > > >
> > > > > It is just a comment such as "Since 1.18. Use X
> > > > > <
> > > > >
> > > >
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > > > >XX
> > > > > instead.". And we can then look it up in the deprecated list[1] in
> > each
> > > > > release and see which method should / can be deprecated.
> > > > >
> > > > > *"At least 1 patch release for the affected minor release for
> > > > > > Experimental APIs"*
> > > > > > The rule is absolutely right. However, afaiac, deprecation is
> > > different
> > > > > as
> > > > > > modification. As a user/dev, I would appreciate, if I do not need
> > to
> > > do
> > > > > any
> > > > > > migration work for any deprecated API between patch releases
> > upgrade.
> > > > > BTW,
> > > > > > if experimental APIs are allowed to change between patches, could
> > we
> > > > just
> > > > > > change them instead of marking them as deprecated and create new
> > ones
> > > > to
> > > > > > replace them?
> > > > >
> > > > > Deprecating an API is just a more elegant way of replacing an API
> > with
> > > a
> > > > > new one. The only difference between the two is whether the old API
> > is
> > > > kept
> > > > > and coexists with the new API for some releases or not. For end
> > users,
> > > > > deprecation-then-remove is much more friendly than direct
> > replacement.
> > > > >
> > > > > 1. How to make sure the new APIs cover all functionality, i.e.
> > backward
> > > > > > compatible, before removing the deprecated APIs? Since the
> > > > > > functionalities could only be built with the new APIs
> iteratively,
> > > > there
> > > > > > will be a while (might be longer than the migration period) that
> > the
> > > > new
> > > > > > APIs are not backward compatible with the deprecated ones.
> > > > >
> > > > > This is orthogonal to the deprecation process, and may not even be
> > > > required
> > > > > in some cases if the function changes by itself. But in general,
> this
> > > > > relies on the developer to decide. A simple test on readiness is to
> > see
> > > > if
> > > > > all the UT / IT cases written with the old API can be migrated to
> the
> > > new
> > > > > one and still work.  If the new API is not ready, we probably
> should
> > > not
> > > > > deprecate the old one to begin with.
> > > > >
> > > > > 2. Is it allowed to remove the deprecated APIs after the defined
> > > > migration
> > > > > > period expires while the new APis are still not backward
> > compatible?
> > > > >
> > > > > By "backwards compatible", do you mean functionally equivalent? If
> > the
> > > > new
> > > > > APIs are designed to be not backwards compatible, then removing the
> > > > > deprecated source code is definitely allowed. If we don't think the
> > new
> > > > API
> > > > > is ready to take over the place for the old one, then we should
> wait.
> > > The
> > > > > migration period is the minimum time we have to wait before
> removing
> > > the
> > > > > source code. A longer migration period is OK.
> > > > >
> > > > > 3. For the case of core API upgrade with downstream
> implementations,
> > > e.g.
> > > > > > connectors, What is the feasible deprecation strategy? Option1
> > > > bottom-up:
> > > > > > make sure the downstream implementation is backward compatible
> > before
> > > > > > removing the deprecated core APIs. Option2 top-down: once the
> > > > downstream
> > > > > > implementation of new APIs works fine, we can remove the
> deprecated
> > > > core
> > > > > > APIs after the migration period expires. The implementation of
> the
> > > > > > deprecated APIs will not get any further update in upcoming
> > > releases(it
> > > > > has
> > > > > > been removed). There might be some missing features in the
> > downstream
> > > > > > implementation of new APIs compared to the old implementation.
> Both
> > > > > options
> > > > > > have their own pros and cons.
> > > > >
> > > > > The downstream projects such as connectors in Flink should also
> > follow
> > > > the
> > > > > migration path we tell our users. i.e. If there is a cascading
> > > backwards
> > > > > incompatible change in the connectors due to a backwards
> incompatible
> > > > > change in the core, and as a result a longer migration period is
> > > > required,
> > > > > then I think we should postpone the removal of source code. But in
> > > > general,
> > > > > we should be able to provide the same migration period in the
> > > connectors
> > > > as
> > > > > the flink core, if the connectors are upgraded to the latest
> version
> > of
> > > > > core promptly.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > > > >
> > > > >
> > > > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > > This is by design. Most of these are @Public APIs that we had
> to
> > > > carry
> > > > > > > around until Flink 2.0, because that was the initial guarantee
> > that
> > > > we
> > > > > > > gave people.
> > > > > > >
> > > > > >
> > > > > > True, I knew @Public APIs could not be removed before the next
> > major
> > > > > > release. I meant house cleaning without violation of these
> > > annotations'
> > > > > > design concept. i.e especially cleaning up for @PublicEvolving
> APIs
> > > > since
> > > > > > they are customer-facing. Regular cleaning up with all other
> > > > > @Experimental
> > > > > > and @Internal APIs would be even better, if there might be some
> > APIs
> > > > > marked
> > > > > > as @deprecated.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <
> > chesnay@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > > > One major issue we have, afaiu, is caused by the lack of
> > > > > > > housekeeping/house
> > > > > > > > cleaning, there are many APIs that were marked as deprecated
> a
> > > few
> > > > > > years
> > > > > > > > ago and still don't get removed. Some APIs should be easy to
> > > remove
> > > > > and
> > > > > > > > others will need some more clear rules, like the issue
> > discussed
> > > at
> > > > > > [1].
> > > > > > >
> > > > > > This is by design. Most of these are @Public APIs that we had to
> > > carry
> > > > > > > around until Flink 2.0, because that was the initial guarantee
> > that
> > > > we
> > > > > > > gave people.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > > > > deprecation period for APIs, particularly PublicEvolving ones.
> > > > > > > For Experimental I don't think it'd be a problem if we could
> > change
> > > > > them
> > > > > > > right away,
> > > > > > > but looking back a bit I don't think it hurts us to also
> enforce
> > > some
> > > > > > > deprecation period.
> > > > > > > 1 release for both of these sound fine to me.
> > > > > > >
> > > > > > >
> > > > > > > My major point of contention is the removal of Public APIs
> > between
> > > > > minor
> > > > > > > versions.
> > > > > > > This to me would a major setback towards a simpler upgrade path
> > for
> > > > > > users.
> > > > > > > If these can be removed in minor versions than what even is a
> > major
> > > > > > > release?
> > > > > > > The very definition we have for Public APIs is that they stick
> > > around
> > > > > > > until the next major one.
> > > > > > > Any rule that theoretically allows for breaking changes in
> Public
> > > API
> > > > > in
> > > > > > > every minor release is in my opinion not a viable option.
> > > > > > >
> > > > > > >
> > > > > > > The "carry recent Public APIs forward into the next major
> > release"
> > > > > thing
> > > > > > > seems to presume a linear release history (aka, if 2.0 is
> > released
> > > > > after
> > > > > > > 1.20, then there will be no 1.21), which I doubt will be the
> > case.
> > > > The
> > > > > > > idea behind it is good, but I'd say the right conclusion would
> be
> > > to
> > > > > not
> > > > > > > make that API public if we know a new major release hits in 3
> > > months
> > > > > and
> > > > > > > is about to modify it. With a regular schedule for major
> releases
> > > > this
> > > > > > > wouldn't be difficult to do.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Matthias Pohl <ma...@aiven.io.INVALID>.
One (made-up) example from the top of my head would have been that the
community decides to focus fully on Kubernetes without considering Yarn
anymore because of some must-have feature on the Kubernetes side. At the
same time there are still some users for whom it would be tricky to migrate
from Yarn to Kubernetes. Therefore, there would be some desire to still
maintain the older major version of Flink that supports Yarn.

But now that I'm thinking about the example, I realize: A discussion on how
we would handle the two major versions in that case, could be started when
we actually run into this issue. It shouldn't be too hard to migrate to a
non-linear versioning from what you are proposing if such a scenario comes
up.

And on the topic of downgrading from @Public to @PublicEvolving:

Demoting a Public API to PublicEvolving API sounds hacky. From what I
> understand the stability guarantee is not revocable because users will rely
> on the stability guarantee to plan their usage of Flink. Demoting an API
> essentially defeats the very purpose of stability guarantee to begin with.
>

I understand the stability guarantee in a way that it only applies within a
major version. Downgrading the stability constraint for an API with a new
major version still seems to comply with the definition of a @Public
annotation as it's similar to changing the API in other ways. But I'm not
insisting on that approach. I just found it a reasonable workaround.

Thanks,
Matthias

On Wed, Jun 14, 2023 at 11:38 AM Becket Qin <be...@gmail.com> wrote:

> Hi Matthias,
>
> Thanks for the feedback.
>
> Do you have an example of behavioral change in mind? Not sure I fully
> understand the concern for behavioral change here. From what I understand,
> any user sensible change in an existing API, regardless of its kind (API
> signature or behavioral change), can always be done in the following way:
>
> 1. Introduce a new API (new package, new class/interface, new method, new
> config, new metric, etc) while marking the old one as deprecated.
> 2. Let the new API and deprecated API coexist for the migration period to
> allow planned migration from the users.
> 3. Remove the deprecated API.
>
> For example, Kafka deprecated its old consumer and replaced it with a new
> Consumer - basically everything changes. The source code of the old
> consumer was kept there for a few years across multiple major versions.
> This does mean we have to keep both of the APIs for a few releases, and
> even fix bugs in the old consumer, so additional maintenance effort is
> required. But this allows the users to keep up with Kafka releases which is
> extremely rewarding.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
> <ma...@aiven.io.invalid> wrote:
>
> > Thanks for starting this discussion, Becket. A few good points were
> raised.
> > Here's what I want to add:
> >
> > Stefan raised the point of behavioral stability (in contrast to API
> > stability). That might be a reason for users to not be able to go ahead
> > with a major version bump. Working around behavioral changes might be
> > trickier than just switching from deprecated to newer APIs. I see your
> > motivation of having a more linear versioning even between major versions
> > to avoid backports. Backports are painful enough for minor versions.
> >
> > But with major versions becoming a thing in the Flink cosmos, I could
> > imagine that the behavioral stability Stefan mentions actually could
> become
> > a bigger issue: Major versions down the road might include bigger
> > behavioral changes which would prevent users from going ahead with the
> > major version bump. I understand that this is out of the original scope
> of
> > this FLIP. But nevertheless, it does support Chesnay's concerns that a
> > linear versioning without maintaining older major versions might not be
> > feasible. It sounds like we should have a discussion about how we treat
> > older major versions here (or have a dedicated discussion on that topic
> > before going ahead with that FLIP).
> >
> > On another note: I like Xintong's proposal of downgrading an API
> > from @Public to @PublicEvolving in the new major version. That would
> allow
> > us to keep the original intention of the @Public annotation alive (i.e.
> > that those APIs are only removed in the next major version).
> >
> > Matthias
> >
> > On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > Thanks for bringing up this discussion, Becket.
> > >
> > > My two cents:
> > >
> > > 1. Do we allow deprecation & removal of APIs without adding a new one
> as
> > a
> > > replacement? The examples in the table give me an impression that
> marking
> > > an API as `@Deprecated` should only happen at the same time of
> > introducing
> > > a new replacing API, which I think is true in most but not all the
> cases.
> > >
> > > If there is a major version bump before 2 minor releases in the current
> > > > major version are reached, the major version should keep the source
> > code
> > > in
> > > > its own minor version until two minor versions are reached. For
> > example,
> > > in
> > > > the above case, if Flink 2.0 is released after 1.20, then the
> > deprecated
> > > > source code of foo will be kept in 2.0 and all the 2.x versions. It
> can
> > > > only be removed in 3.0.
> > > >
> > >
> > > 2. I think this might be a bit too strict. For an API that we already
> > > decided to remove, having to keep it for all the 2.x versions simply
> > > because there's less than 2 minor releases between making the decision
> > and
> > > the major release bump seems not necessarily. Alternatively, I'd like
> to
> > > propose to remove the `@Public` annotation (or downgrade it to
> > > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > Thanks for the feedback. Please see the answers to your questions
> > below:
> > > >
> > > > *"Always add a "Since X.X.X" comment to indicate when was a class /
> > > > > interface / method marked as deprecated."*
> > > > >  Could you describe it with a code example? Do you mean Java
> > comments?
> > > >
> > > > It is just a comment such as "Since 1.18. Use X
> > > > <
> > > >
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > > >XX
> > > > instead.". And we can then look it up in the deprecated list[1] in
> each
> > > > release and see which method should / can be deprecated.
> > > >
> > > > *"At least 1 patch release for the affected minor release for
> > > > > Experimental APIs"*
> > > > > The rule is absolutely right. However, afaiac, deprecation is
> > different
> > > > as
> > > > > modification. As a user/dev, I would appreciate, if I do not need
> to
> > do
> > > > any
> > > > > migration work for any deprecated API between patch releases
> upgrade.
> > > > BTW,
> > > > > if experimental APIs are allowed to change between patches, could
> we
> > > just
> > > > > change them instead of marking them as deprecated and create new
> ones
> > > to
> > > > > replace them?
> > > >
> > > > Deprecating an API is just a more elegant way of replacing an API
> with
> > a
> > > > new one. The only difference between the two is whether the old API
> is
> > > kept
> > > > and coexists with the new API for some releases or not. For end
> users,
> > > > deprecation-then-remove is much more friendly than direct
> replacement.
> > > >
> > > > 1. How to make sure the new APIs cover all functionality, i.e.
> backward
> > > > > compatible, before removing the deprecated APIs? Since the
> > > > > functionalities could only be built with the new APIs iteratively,
> > > there
> > > > > will be a while (might be longer than the migration period) that
> the
> > > new
> > > > > APIs are not backward compatible with the deprecated ones.
> > > >
> > > > This is orthogonal to the deprecation process, and may not even be
> > > required
> > > > in some cases if the function changes by itself. But in general, this
> > > > relies on the developer to decide. A simple test on readiness is to
> see
> > > if
> > > > all the UT / IT cases written with the old API can be migrated to the
> > new
> > > > one and still work.  If the new API is not ready, we probably should
> > not
> > > > deprecate the old one to begin with.
> > > >
> > > > 2. Is it allowed to remove the deprecated APIs after the defined
> > > migration
> > > > > period expires while the new APis are still not backward
> compatible?
> > > >
> > > > By "backwards compatible", do you mean functionally equivalent? If
> the
> > > new
> > > > APIs are designed to be not backwards compatible, then removing the
> > > > deprecated source code is definitely allowed. If we don't think the
> new
> > > API
> > > > is ready to take over the place for the old one, then we should wait.
> > The
> > > > migration period is the minimum time we have to wait before removing
> > the
> > > > source code. A longer migration period is OK.
> > > >
> > > > 3. For the case of core API upgrade with downstream implementations,
> > e.g.
> > > > > connectors, What is the feasible deprecation strategy? Option1
> > > bottom-up:
> > > > > make sure the downstream implementation is backward compatible
> before
> > > > > removing the deprecated core APIs. Option2 top-down: once the
> > > downstream
> > > > > implementation of new APIs works fine, we can remove the deprecated
> > > core
> > > > > APIs after the migration period expires. The implementation of the
> > > > > deprecated APIs will not get any further update in upcoming
> > releases(it
> > > > has
> > > > > been removed). There might be some missing features in the
> downstream
> > > > > implementation of new APIs compared to the old implementation. Both
> > > > options
> > > > > have their own pros and cons.
> > > >
> > > > The downstream projects such as connectors in Flink should also
> follow
> > > the
> > > > migration path we tell our users. i.e. If there is a cascading
> > backwards
> > > > incompatible change in the connectors due to a backwards incompatible
> > > > change in the core, and as a result a longer migration period is
> > > required,
> > > > then I think we should postpone the removal of source code. But in
> > > general,
> > > > we should be able to provide the same migration period in the
> > connectors
> > > as
> > > > the flink core, if the connectors are upgraded to the latest version
> of
> > > > core promptly.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > > >
> > > >
> > > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > > This is by design. Most of these are @Public APIs that we had to
> > > carry
> > > > > > around until Flink 2.0, because that was the initial guarantee
> that
> > > we
> > > > > > gave people.
> > > > > >
> > > > >
> > > > > True, I knew @Public APIs could not be removed before the next
> major
> > > > > release. I meant house cleaning without violation of these
> > annotations'
> > > > > design concept. i.e especially cleaning up for @PublicEvolving APIs
> > > since
> > > > > they are customer-facing. Regular cleaning up with all other
> > > > @Experimental
> > > > > and @Internal APIs would be even better, if there might be some
> APIs
> > > > marked
> > > > > as @deprecated.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <
> chesnay@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > > One major issue we have, afaiu, is caused by the lack of
> > > > > > housekeeping/house
> > > > > > > cleaning, there are many APIs that were marked as deprecated a
> > few
> > > > > years
> > > > > > > ago and still don't get removed. Some APIs should be easy to
> > remove
> > > > and
> > > > > > > others will need some more clear rules, like the issue
> discussed
> > at
> > > > > [1].
> > > > > >
> > > > > This is by design. Most of these are @Public APIs that we had to
> > carry
> > > > > > around until Flink 2.0, because that was the initial guarantee
> that
> > > we
> > > > > > gave people.
> > > > >
> > > > >
> > > > > >
> > > > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > > > deprecation period for APIs, particularly PublicEvolving ones.
> > > > > > For Experimental I don't think it'd be a problem if we could
> change
> > > > them
> > > > > > right away,
> > > > > > but looking back a bit I don't think it hurts us to also enforce
> > some
> > > > > > deprecation period.
> > > > > > 1 release for both of these sound fine to me.
> > > > > >
> > > > > >
> > > > > > My major point of contention is the removal of Public APIs
> between
> > > > minor
> > > > > > versions.
> > > > > > This to me would a major setback towards a simpler upgrade path
> for
> > > > > users.
> > > > > > If these can be removed in minor versions than what even is a
> major
> > > > > > release?
> > > > > > The very definition we have for Public APIs is that they stick
> > around
> > > > > > until the next major one.
> > > > > > Any rule that theoretically allows for breaking changes in Public
> > API
> > > > in
> > > > > > every minor release is in my opinion not a viable option.
> > > > > >
> > > > > >
> > > > > > The "carry recent Public APIs forward into the next major
> release"
> > > > thing
> > > > > > seems to presume a linear release history (aka, if 2.0 is
> released
> > > > after
> > > > > > 1.20, then there will be no 1.21), which I doubt will be the
> case.
> > > The
> > > > > > idea behind it is good, but I'd say the right conclusion would be
> > to
> > > > not
> > > > > > make that API public if we know a new major release hits in 3
> > months
> > > > and
> > > > > > is about to modify it. With a regular schedule for major releases
> > > this
> > > > > > wouldn't be difficult to do.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Matthias,

Thanks for the feedback.

Do you have an example of behavioral change in mind? Not sure I fully
understand the concern for behavioral change here. From what I understand,
any user sensible change in an existing API, regardless of its kind (API
signature or behavioral change), can always be done in the following way:

1. Introduce a new API (new package, new class/interface, new method, new
config, new metric, etc) while marking the old one as deprecated.
2. Let the new API and deprecated API coexist for the migration period to
allow planned migration from the users.
3. Remove the deprecated API.

For example, Kafka deprecated its old consumer and replaced it with a new
Consumer - basically everything changes. The source code of the old
consumer was kept there for a few years across multiple major versions.
This does mean we have to keep both of the APIs for a few releases, and
even fix bugs in the old consumer, so additional maintenance effort is
required. But this allows the users to keep up with Kafka releases which is
extremely rewarding.

Thanks,

Jiangjie (Becket) Qin

On Wed, Jun 14, 2023 at 5:06 PM Matthias Pohl
<ma...@aiven.io.invalid> wrote:

> Thanks for starting this discussion, Becket. A few good points were raised.
> Here's what I want to add:
>
> Stefan raised the point of behavioral stability (in contrast to API
> stability). That might be a reason for users to not be able to go ahead
> with a major version bump. Working around behavioral changes might be
> trickier than just switching from deprecated to newer APIs. I see your
> motivation of having a more linear versioning even between major versions
> to avoid backports. Backports are painful enough for minor versions.
>
> But with major versions becoming a thing in the Flink cosmos, I could
> imagine that the behavioral stability Stefan mentions actually could become
> a bigger issue: Major versions down the road might include bigger
> behavioral changes which would prevent users from going ahead with the
> major version bump. I understand that this is out of the original scope of
> this FLIP. But nevertheless, it does support Chesnay's concerns that a
> linear versioning without maintaining older major versions might not be
> feasible. It sounds like we should have a discussion about how we treat
> older major versions here (or have a dedicated discussion on that topic
> before going ahead with that FLIP).
>
> On another note: I like Xintong's proposal of downgrading an API
> from @Public to @PublicEvolving in the new major version. That would allow
> us to keep the original intention of the @Public annotation alive (i.e.
> that those APIs are only removed in the next major version).
>
> Matthias
>
> On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Thanks for bringing up this discussion, Becket.
> >
> > My two cents:
> >
> > 1. Do we allow deprecation & removal of APIs without adding a new one as
> a
> > replacement? The examples in the table give me an impression that marking
> > an API as `@Deprecated` should only happen at the same time of
> introducing
> > a new replacing API, which I think is true in most but not all the cases.
> >
> > If there is a major version bump before 2 minor releases in the current
> > > major version are reached, the major version should keep the source
> code
> > in
> > > its own minor version until two minor versions are reached. For
> example,
> > in
> > > the above case, if Flink 2.0 is released after 1.20, then the
> deprecated
> > > source code of foo will be kept in 2.0 and all the 2.x versions. It can
> > > only be removed in 3.0.
> > >
> >
> > 2. I think this might be a bit too strict. For an API that we already
> > decided to remove, having to keep it for all the 2.x versions simply
> > because there's less than 2 minor releases between making the decision
> and
> > the major release bump seems not necessarily. Alternatively, I'd like to
> > propose to remove the `@Public` annotation (or downgrade it to
> > `@PublicEvolving`) in 2.0, and remove it in 2.2.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for the feedback. Please see the answers to your questions
> below:
> > >
> > > *"Always add a "Since X.X.X" comment to indicate when was a class /
> > > > interface / method marked as deprecated."*
> > > >  Could you describe it with a code example? Do you mean Java
> comments?
> > >
> > > It is just a comment such as "Since 1.18. Use X
> > > <
> > >
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > > >XX
> > > instead.". And we can then look it up in the deprecated list[1] in each
> > > release and see which method should / can be deprecated.
> > >
> > > *"At least 1 patch release for the affected minor release for
> > > > Experimental APIs"*
> > > > The rule is absolutely right. However, afaiac, deprecation is
> different
> > > as
> > > > modification. As a user/dev, I would appreciate, if I do not need to
> do
> > > any
> > > > migration work for any deprecated API between patch releases upgrade.
> > > BTW,
> > > > if experimental APIs are allowed to change between patches, could we
> > just
> > > > change them instead of marking them as deprecated and create new ones
> > to
> > > > replace them?
> > >
> > > Deprecating an API is just a more elegant way of replacing an API with
> a
> > > new one. The only difference between the two is whether the old API is
> > kept
> > > and coexists with the new API for some releases or not. For end users,
> > > deprecation-then-remove is much more friendly than direct replacement.
> > >
> > > 1. How to make sure the new APIs cover all functionality, i.e. backward
> > > > compatible, before removing the deprecated APIs? Since the
> > > > functionalities could only be built with the new APIs iteratively,
> > there
> > > > will be a while (might be longer than the migration period) that the
> > new
> > > > APIs are not backward compatible with the deprecated ones.
> > >
> > > This is orthogonal to the deprecation process, and may not even be
> > required
> > > in some cases if the function changes by itself. But in general, this
> > > relies on the developer to decide. A simple test on readiness is to see
> > if
> > > all the UT / IT cases written with the old API can be migrated to the
> new
> > > one and still work.  If the new API is not ready, we probably should
> not
> > > deprecate the old one to begin with.
> > >
> > > 2. Is it allowed to remove the deprecated APIs after the defined
> > migration
> > > > period expires while the new APis are still not backward compatible?
> > >
> > > By "backwards compatible", do you mean functionally equivalent? If the
> > new
> > > APIs are designed to be not backwards compatible, then removing the
> > > deprecated source code is definitely allowed. If we don't think the new
> > API
> > > is ready to take over the place for the old one, then we should wait.
> The
> > > migration period is the minimum time we have to wait before removing
> the
> > > source code. A longer migration period is OK.
> > >
> > > 3. For the case of core API upgrade with downstream implementations,
> e.g.
> > > > connectors, What is the feasible deprecation strategy? Option1
> > bottom-up:
> > > > make sure the downstream implementation is backward compatible before
> > > > removing the deprecated core APIs. Option2 top-down: once the
> > downstream
> > > > implementation of new APIs works fine, we can remove the deprecated
> > core
> > > > APIs after the migration period expires. The implementation of the
> > > > deprecated APIs will not get any further update in upcoming
> releases(it
> > > has
> > > > been removed). There might be some missing features in the downstream
> > > > implementation of new APIs compared to the old implementation. Both
> > > options
> > > > have their own pros and cons.
> > >
> > > The downstream projects such as connectors in Flink should also follow
> > the
> > > migration path we tell our users. i.e. If there is a cascading
> backwards
> > > incompatible change in the connectors due to a backwards incompatible
> > > change in the core, and as a result a longer migration period is
> > required,
> > > then I think we should postpone the removal of source code. But in
> > general,
> > > we should be able to provide the same migration period in the
> connectors
> > as
> > > the flink core, if the connectors are upgraded to the latest version of
> > > core promptly.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > [1]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> > >
> > >
> > > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > > This is by design. Most of these are @Public APIs that we had to
> > carry
> > > > > around until Flink 2.0, because that was the initial guarantee that
> > we
> > > > > gave people.
> > > > >
> > > >
> > > > True, I knew @Public APIs could not be removed before the next major
> > > > release. I meant house cleaning without violation of these
> annotations'
> > > > design concept. i.e especially cleaning up for @PublicEvolving APIs
> > since
> > > > they are customer-facing. Regular cleaning up with all other
> > > @Experimental
> > > > and @Internal APIs would be even better, if there might be some APIs
> > > marked
> > > > as @deprecated.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > >
> > > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <chesnay@apache.org
> >
> > > > wrote:
> > > >
> > > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > > One major issue we have, afaiu, is caused by the lack of
> > > > > housekeeping/house
> > > > > > cleaning, there are many APIs that were marked as deprecated a
> few
> > > > years
> > > > > > ago and still don't get removed. Some APIs should be easy to
> remove
> > > and
> > > > > > others will need some more clear rules, like the issue discussed
> at
> > > > [1].
> > > > >
> > > > This is by design. Most of these are @Public APIs that we had to
> carry
> > > > > around until Flink 2.0, because that was the initial guarantee that
> > we
> > > > > gave people.
> > > >
> > > >
> > > > >
> > > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > > deprecation period for APIs, particularly PublicEvolving ones.
> > > > > For Experimental I don't think it'd be a problem if we could change
> > > them
> > > > > right away,
> > > > > but looking back a bit I don't think it hurts us to also enforce
> some
> > > > > deprecation period.
> > > > > 1 release for both of these sound fine to me.
> > > > >
> > > > >
> > > > > My major point of contention is the removal of Public APIs between
> > > minor
> > > > > versions.
> > > > > This to me would a major setback towards a simpler upgrade path for
> > > > users.
> > > > > If these can be removed in minor versions than what even is a major
> > > > > release?
> > > > > The very definition we have for Public APIs is that they stick
> around
> > > > > until the next major one.
> > > > > Any rule that theoretically allows for breaking changes in Public
> API
> > > in
> > > > > every minor release is in my opinion not a viable option.
> > > > >
> > > > >
> > > > > The "carry recent Public APIs forward into the next major release"
> > > thing
> > > > > seems to presume a linear release history (aka, if 2.0 is released
> > > after
> > > > > 1.20, then there will be no 1.21), which I doubt will be the case.
> > The
> > > > > idea behind it is good, but I'd say the right conclusion would be
> to
> > > not
> > > > > make that API public if we know a new major release hits in 3
> months
> > > and
> > > > > is about to modify it. With a regular schedule for major releases
> > this
> > > > > wouldn't be difficult to do.
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Matthias Pohl <ma...@aiven.io.INVALID>.
Thanks for starting this discussion, Becket. A few good points were raised.
Here's what I want to add:

Stefan raised the point of behavioral stability (in contrast to API
stability). That might be a reason for users to not be able to go ahead
with a major version bump. Working around behavioral changes might be
trickier than just switching from deprecated to newer APIs. I see your
motivation of having a more linear versioning even between major versions
to avoid backports. Backports are painful enough for minor versions.

But with major versions becoming a thing in the Flink cosmos, I could
imagine that the behavioral stability Stefan mentions actually could become
a bigger issue: Major versions down the road might include bigger
behavioral changes which would prevent users from going ahead with the
major version bump. I understand that this is out of the original scope of
this FLIP. But nevertheless, it does support Chesnay's concerns that a
linear versioning without maintaining older major versions might not be
feasible. It sounds like we should have a discussion about how we treat
older major versions here (or have a dedicated discussion on that topic
before going ahead with that FLIP).

On another note: I like Xintong's proposal of downgrading an API
from @Public to @PublicEvolving in the new major version. That would allow
us to keep the original intention of the @Public annotation alive (i.e.
that those APIs are only removed in the next major version).

Matthias

On Wed, Jun 14, 2023 at 10:10 AM Xintong Song <to...@gmail.com> wrote:

> Thanks for bringing up this discussion, Becket.
>
> My two cents:
>
> 1. Do we allow deprecation & removal of APIs without adding a new one as a
> replacement? The examples in the table give me an impression that marking
> an API as `@Deprecated` should only happen at the same time of introducing
> a new replacing API, which I think is true in most but not all the cases.
>
> If there is a major version bump before 2 minor releases in the current
> > major version are reached, the major version should keep the source code
> in
> > its own minor version until two minor versions are reached. For example,
> in
> > the above case, if Flink 2.0 is released after 1.20, then the deprecated
> > source code of foo will be kept in 2.0 and all the 2.x versions. It can
> > only be removed in 3.0.
> >
>
> 2. I think this might be a bit too strict. For an API that we already
> decided to remove, having to keep it for all the 2.x versions simply
> because there's less than 2 minor releases between making the decision and
> the major release bump seems not necessarily. Alternatively, I'd like to
> propose to remove the `@Public` annotation (or downgrade it to
> `@PublicEvolving`) in 2.0, and remove it in 2.2.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > Thanks for the feedback. Please see the answers to your questions below:
> >
> > *"Always add a "Since X.X.X" comment to indicate when was a class /
> > > interface / method marked as deprecated."*
> > >  Could you describe it with a code example? Do you mean Java comments?
> >
> > It is just a comment such as "Since 1.18. Use X
> > <
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > >XX
> > instead.". And we can then look it up in the deprecated list[1] in each
> > release and see which method should / can be deprecated.
> >
> > *"At least 1 patch release for the affected minor release for
> > > Experimental APIs"*
> > > The rule is absolutely right. However, afaiac, deprecation is different
> > as
> > > modification. As a user/dev, I would appreciate, if I do not need to do
> > any
> > > migration work for any deprecated API between patch releases upgrade.
> > BTW,
> > > if experimental APIs are allowed to change between patches, could we
> just
> > > change them instead of marking them as deprecated and create new ones
> to
> > > replace them?
> >
> > Deprecating an API is just a more elegant way of replacing an API with a
> > new one. The only difference between the two is whether the old API is
> kept
> > and coexists with the new API for some releases or not. For end users,
> > deprecation-then-remove is much more friendly than direct replacement.
> >
> > 1. How to make sure the new APIs cover all functionality, i.e. backward
> > > compatible, before removing the deprecated APIs? Since the
> > > functionalities could only be built with the new APIs iteratively,
> there
> > > will be a while (might be longer than the migration period) that the
> new
> > > APIs are not backward compatible with the deprecated ones.
> >
> > This is orthogonal to the deprecation process, and may not even be
> required
> > in some cases if the function changes by itself. But in general, this
> > relies on the developer to decide. A simple test on readiness is to see
> if
> > all the UT / IT cases written with the old API can be migrated to the new
> > one and still work.  If the new API is not ready, we probably should not
> > deprecate the old one to begin with.
> >
> > 2. Is it allowed to remove the deprecated APIs after the defined
> migration
> > > period expires while the new APis are still not backward compatible?
> >
> > By "backwards compatible", do you mean functionally equivalent? If the
> new
> > APIs are designed to be not backwards compatible, then removing the
> > deprecated source code is definitely allowed. If we don't think the new
> API
> > is ready to take over the place for the old one, then we should wait. The
> > migration period is the minimum time we have to wait before removing the
> > source code. A longer migration period is OK.
> >
> > 3. For the case of core API upgrade with downstream implementations, e.g.
> > > connectors, What is the feasible deprecation strategy? Option1
> bottom-up:
> > > make sure the downstream implementation is backward compatible before
> > > removing the deprecated core APIs. Option2 top-down: once the
> downstream
> > > implementation of new APIs works fine, we can remove the deprecated
> core
> > > APIs after the migration period expires. The implementation of the
> > > deprecated APIs will not get any further update in upcoming releases(it
> > has
> > > been removed). There might be some missing features in the downstream
> > > implementation of new APIs compared to the old implementation. Both
> > options
> > > have their own pros and cons.
> >
> > The downstream projects such as connectors in Flink should also follow
> the
> > migration path we tell our users. i.e. If there is a cascading backwards
> > incompatible change in the connectors due to a backwards incompatible
> > change in the core, and as a result a longer migration period is
> required,
> > then I think we should postpone the removal of source code. But in
> general,
> > we should be able to provide the same migration period in the connectors
> as
> > the flink core, if the connectors are upgraded to the latest version of
> > core promptly.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> >
> >
> > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > > This is by design. Most of these are @Public APIs that we had to
> carry
> > > > around until Flink 2.0, because that was the initial guarantee that
> we
> > > > gave people.
> > > >
> > >
> > > True, I knew @Public APIs could not be removed before the next major
> > > release. I meant house cleaning without violation of these annotations'
> > > design concept. i.e especially cleaning up for @PublicEvolving APIs
> since
> > > they are customer-facing. Regular cleaning up with all other
> > @Experimental
> > > and @Internal APIs would be even better, if there might be some APIs
> > marked
> > > as @deprecated.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > >
> > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org>
> > > wrote:
> > >
> > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > One major issue we have, afaiu, is caused by the lack of
> > > > housekeeping/house
> > > > > cleaning, there are many APIs that were marked as deprecated a few
> > > years
> > > > > ago and still don't get removed. Some APIs should be easy to remove
> > and
> > > > > others will need some more clear rules, like the issue discussed at
> > > [1].
> > > >
> > > This is by design. Most of these are @Public APIs that we had to carry
> > > > around until Flink 2.0, because that was the initial guarantee that
> we
> > > > gave people.
> > >
> > >
> > > >
> > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > deprecation period for APIs, particularly PublicEvolving ones.
> > > > For Experimental I don't think it'd be a problem if we could change
> > them
> > > > right away,
> > > > but looking back a bit I don't think it hurts us to also enforce some
> > > > deprecation period.
> > > > 1 release for both of these sound fine to me.
> > > >
> > > >
> > > > My major point of contention is the removal of Public APIs between
> > minor
> > > > versions.
> > > > This to me would a major setback towards a simpler upgrade path for
> > > users.
> > > > If these can be removed in minor versions than what even is a major
> > > > release?
> > > > The very definition we have for Public APIs is that they stick around
> > > > until the next major one.
> > > > Any rule that theoretically allows for breaking changes in Public API
> > in
> > > > every minor release is in my opinion not a viable option.
> > > >
> > > >
> > > > The "carry recent Public APIs forward into the next major release"
> > thing
> > > > seems to presume a linear release history (aka, if 2.0 is released
> > after
> > > > 1.20, then there will be no 1.21), which I doubt will be the case.
> The
> > > > idea behind it is good, but I'd say the right conclusion would be to
> > not
> > > > make that API public if we know a new major release hits in 3 months
> > and
> > > > is about to modify it. With a regular schedule for major releases
> this
> > > > wouldn't be difficult to do.
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

Thanks for the comment. Please see the replies below:

1. Do we allow deprecation & removal of APIs without adding a new one as a
> replacement? The examples in the table give me an impression that marking
> an API as `@Deprecated` should only happen at the same time of introducing
> a new replacing API, which I think is true in most but not all the cases.

Right, it is not necessary to have a replacement for the deprecated API, if
we decide to sunset the functionality. That does not change the migration
period, though.

2. I think this might be a bit too strict. For an API that we already
> decided to remove, having to keep it for all the 2.x versions simply
> because there's less than 2 minor releases between making the decision and
> the major release bump seems not necessarily. Alternatively, I'd like to
> propose to remove the `@Public` annotation (or downgrade it to
> `@PublicEvolving`) in 2.0, and remove it in 2.2.

I am not sure this is a good practice. The purpose of the migration period
is to give users enough time to adapt to a breaking API change without
holding them back from upgrading Flink. The reason we say Public API needs
at least two minor releases is because there are probably more users picked
them up over time and more jobs are running using these APIs. So, the
public APIs just require a larger migration window. Admittedly, this will
introduce a higher maintenance cost for us, this is why Public APIs should
be treated seriously. If the promotion of a PublicEvolving API to a Public
API requires two minor version releases, deprecation of a Public API should
only take longer.

Demoting a Public API to PublicEvolving API sounds hacky. From what I
understand the stability guarantee is not revocable because users will rely
on the stability guarantee to plan their usage of Flink. Demoting an API
essentially defeats the very purpose of stability guarantee to begin with.

If the concern of keeping a migration period of two minor releases across
major versions is about the maintenance overhead, we can choose to bump up
the major version to 3.0 at some point after the migration period has
passed, assuming by then most of the users have migrated away from the
deprecated Public API.

Thanks,

Jiangjie (Becket) Qin


On Wed, Jun 14, 2023 at 4:10 PM Xintong Song <to...@gmail.com> wrote:

> Thanks for bringing up this discussion, Becket.
>
> My two cents:
>
> 1. Do we allow deprecation & removal of APIs without adding a new one as a
> replacement? The examples in the table give me an impression that marking
> an API as `@Deprecated` should only happen at the same time of introducing
> a new replacing API, which I think is true in most but not all the cases.
>
> If there is a major version bump before 2 minor releases in the current
> > major version are reached, the major version should keep the source code
> in
> > its own minor version until two minor versions are reached. For example,
> in
> > the above case, if Flink 2.0 is released after 1.20, then the deprecated
> > source code of foo will be kept in 2.0 and all the 2.x versions. It can
> > only be removed in 3.0.
> >
>
> 2. I think this might be a bit too strict. For an API that we already
> decided to remove, having to keep it for all the 2.x versions simply
> because there's less than 2 minor releases between making the decision and
> the major release bump seems not necessarily. Alternatively, I'd like to
> propose to remove the `@Public` annotation (or downgrade it to
> `@PublicEvolving`) in 2.0, and remove it in 2.2.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > Thanks for the feedback. Please see the answers to your questions below:
> >
> > *"Always add a "Since X.X.X" comment to indicate when was a class /
> > > interface / method marked as deprecated."*
> > >  Could you describe it with a code example? Do you mean Java comments?
> >
> > It is just a comment such as "Since 1.18. Use X
> > <
> >
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> > >XX
> > instead.". And we can then look it up in the deprecated list[1] in each
> > release and see which method should / can be deprecated.
> >
> > *"At least 1 patch release for the affected minor release for
> > > Experimental APIs"*
> > > The rule is absolutely right. However, afaiac, deprecation is different
> > as
> > > modification. As a user/dev, I would appreciate, if I do not need to do
> > any
> > > migration work for any deprecated API between patch releases upgrade.
> > BTW,
> > > if experimental APIs are allowed to change between patches, could we
> just
> > > change them instead of marking them as deprecated and create new ones
> to
> > > replace them?
> >
> > Deprecating an API is just a more elegant way of replacing an API with a
> > new one. The only difference between the two is whether the old API is
> kept
> > and coexists with the new API for some releases or not. For end users,
> > deprecation-then-remove is much more friendly than direct replacement.
> >
> > 1. How to make sure the new APIs cover all functionality, i.e. backward
> > > compatible, before removing the deprecated APIs? Since the
> > > functionalities could only be built with the new APIs iteratively,
> there
> > > will be a while (might be longer than the migration period) that the
> new
> > > APIs are not backward compatible with the deprecated ones.
> >
> > This is orthogonal to the deprecation process, and may not even be
> required
> > in some cases if the function changes by itself. But in general, this
> > relies on the developer to decide. A simple test on readiness is to see
> if
> > all the UT / IT cases written with the old API can be migrated to the new
> > one and still work.  If the new API is not ready, we probably should not
> > deprecate the old one to begin with.
> >
> > 2. Is it allowed to remove the deprecated APIs after the defined
> migration
> > > period expires while the new APis are still not backward compatible?
> >
> > By "backwards compatible", do you mean functionally equivalent? If the
> new
> > APIs are designed to be not backwards compatible, then removing the
> > deprecated source code is definitely allowed. If we don't think the new
> API
> > is ready to take over the place for the old one, then we should wait. The
> > migration period is the minimum time we have to wait before removing the
> > source code. A longer migration period is OK.
> >
> > 3. For the case of core API upgrade with downstream implementations, e.g.
> > > connectors, What is the feasible deprecation strategy? Option1
> bottom-up:
> > > make sure the downstream implementation is backward compatible before
> > > removing the deprecated core APIs. Option2 top-down: once the
> downstream
> > > implementation of new APIs works fine, we can remove the deprecated
> core
> > > APIs after the migration period expires. The implementation of the
> > > deprecated APIs will not get any further update in upcoming releases(it
> > has
> > > been removed). There might be some missing features in the downstream
> > > implementation of new APIs compared to the old implementation. Both
> > options
> > > have their own pros and cons.
> >
> > The downstream projects such as connectors in Flink should also follow
> the
> > migration path we tell our users. i.e. If there is a cascading backwards
> > incompatible change in the connectors due to a backwards incompatible
> > change in the core, and as a result a longer migration period is
> required,
> > then I think we should postpone the removal of source code. But in
> general,
> > we should be able to provide the same migration period in the connectors
> as
> > the flink core, if the connectors are upgraded to the latest version of
> > core promptly.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
> >
> >
> > On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > > This is by design. Most of these are @Public APIs that we had to
> carry
> > > > around until Flink 2.0, because that was the initial guarantee that
> we
> > > > gave people.
> > > >
> > >
> > > True, I knew @Public APIs could not be removed before the next major
> > > release. I meant house cleaning without violation of these annotations'
> > > design concept. i.e especially cleaning up for @PublicEvolving APIs
> since
> > > they are customer-facing. Regular cleaning up with all other
> > @Experimental
> > > and @Internal APIs would be even better, if there might be some APIs
> > marked
> > > as @deprecated.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > >
> > > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org>
> > > wrote:
> > >
> > > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > > One major issue we have, afaiu, is caused by the lack of
> > > > housekeeping/house
> > > > > cleaning, there are many APIs that were marked as deprecated a few
> > > years
> > > > > ago and still don't get removed. Some APIs should be easy to remove
> > and
> > > > > others will need some more clear rules, like the issue discussed at
> > > [1].
> > > >
> > > This is by design. Most of these are @Public APIs that we had to carry
> > > > around until Flink 2.0, because that was the initial guarantee that
> we
> > > > gave people.
> > >
> > >
> > > >
> > > > As for the FLIP, I like the idea of explicitly writing down a
> > > > deprecation period for APIs, particularly PublicEvolving ones.
> > > > For Experimental I don't think it'd be a problem if we could change
> > them
> > > > right away,
> > > > but looking back a bit I don't think it hurts us to also enforce some
> > > > deprecation period.
> > > > 1 release for both of these sound fine to me.
> > > >
> > > >
> > > > My major point of contention is the removal of Public APIs between
> > minor
> > > > versions.
> > > > This to me would a major setback towards a simpler upgrade path for
> > > users.
> > > > If these can be removed in minor versions than what even is a major
> > > > release?
> > > > The very definition we have for Public APIs is that they stick around
> > > > until the next major one.
> > > > Any rule that theoretically allows for breaking changes in Public API
> > in
> > > > every minor release is in my opinion not a viable option.
> > > >
> > > >
> > > > The "carry recent Public APIs forward into the next major release"
> > thing
> > > > seems to presume a linear release history (aka, if 2.0 is released
> > after
> > > > 1.20, then there will be no 1.21), which I doubt will be the case.
> The
> > > > idea behind it is good, but I'd say the right conclusion would be to
> > not
> > > > make that API public if we know a new major release hits in 3 months
> > and
> > > > is about to modify it. With a regular schedule for major releases
> this
> > > > wouldn't be difficult to do.
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Xintong Song <to...@gmail.com>.
Thanks for bringing up this discussion, Becket.

My two cents:

1. Do we allow deprecation & removal of APIs without adding a new one as a
replacement? The examples in the table give me an impression that marking
an API as `@Deprecated` should only happen at the same time of introducing
a new replacing API, which I think is true in most but not all the cases.

If there is a major version bump before 2 minor releases in the current
> major version are reached, the major version should keep the source code in
> its own minor version until two minor versions are reached. For example, in
> the above case, if Flink 2.0 is released after 1.20, then the deprecated
> source code of foo will be kept in 2.0 and all the 2.x versions. It can
> only be removed in 3.0.
>

2. I think this might be a bit too strict. For an API that we already
decided to remove, having to keep it for all the 2.x versions simply
because there's less than 2 minor releases between making the decision and
the major release bump seems not necessarily. Alternatively, I'd like to
propose to remove the `@Public` annotation (or downgrade it to
`@PublicEvolving`) in 2.0, and remove it in 2.2.

Best,

Xintong



On Wed, Jun 14, 2023 at 3:56 PM Becket Qin <be...@gmail.com> wrote:

> Hi Jing,
>
> Thanks for the feedback. Please see the answers to your questions below:
>
> *"Always add a "Since X.X.X" comment to indicate when was a class /
> > interface / method marked as deprecated."*
> >  Could you describe it with a code example? Do you mean Java comments?
>
> It is just a comment such as "Since 1.18. Use X
> <
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> >XX
> instead.". And we can then look it up in the deprecated list[1] in each
> release and see which method should / can be deprecated.
>
> *"At least 1 patch release for the affected minor release for
> > Experimental APIs"*
> > The rule is absolutely right. However, afaiac, deprecation is different
> as
> > modification. As a user/dev, I would appreciate, if I do not need to do
> any
> > migration work for any deprecated API between patch releases upgrade.
> BTW,
> > if experimental APIs are allowed to change between patches, could we just
> > change them instead of marking them as deprecated and create new ones to
> > replace them?
>
> Deprecating an API is just a more elegant way of replacing an API with a
> new one. The only difference between the two is whether the old API is kept
> and coexists with the new API for some releases or not. For end users,
> deprecation-then-remove is much more friendly than direct replacement.
>
> 1. How to make sure the new APIs cover all functionality, i.e. backward
> > compatible, before removing the deprecated APIs? Since the
> > functionalities could only be built with the new APIs iteratively, there
> > will be a while (might be longer than the migration period) that the new
> > APIs are not backward compatible with the deprecated ones.
>
> This is orthogonal to the deprecation process, and may not even be required
> in some cases if the function changes by itself. But in general, this
> relies on the developer to decide. A simple test on readiness is to see if
> all the UT / IT cases written with the old API can be migrated to the new
> one and still work.  If the new API is not ready, we probably should not
> deprecate the old one to begin with.
>
> 2. Is it allowed to remove the deprecated APIs after the defined migration
> > period expires while the new APis are still not backward compatible?
>
> By "backwards compatible", do you mean functionally equivalent? If the new
> APIs are designed to be not backwards compatible, then removing the
> deprecated source code is definitely allowed. If we don't think the new API
> is ready to take over the place for the old one, then we should wait. The
> migration period is the minimum time we have to wait before removing the
> source code. A longer migration period is OK.
>
> 3. For the case of core API upgrade with downstream implementations, e.g.
> > connectors, What is the feasible deprecation strategy? Option1 bottom-up:
> > make sure the downstream implementation is backward compatible before
> > removing the deprecated core APIs. Option2 top-down: once the downstream
> > implementation of new APIs works fine, we can remove the deprecated core
> > APIs after the migration period expires. The implementation of the
> > deprecated APIs will not get any further update in upcoming releases(it
> has
> > been removed). There might be some missing features in the downstream
> > implementation of new APIs compared to the old implementation. Both
> options
> > have their own pros and cons.
>
> The downstream projects such as connectors in Flink should also follow the
> migration path we tell our users. i.e. If there is a cascading backwards
> incompatible change in the connectors due to a backwards incompatible
> change in the core, and as a result a longer migration period is required,
> then I think we should postpone the removal of source code. But in general,
> we should be able to provide the same migration period in the connectors as
> the flink core, if the connectors are upgraded to the latest version of
> core promptly.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
>
>
> On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > > This is by design. Most of these are @Public APIs that we had to carry
> > > around until Flink 2.0, because that was the initial guarantee that we
> > > gave people.
> > >
> >
> > True, I knew @Public APIs could not be removed before the next major
> > release. I meant house cleaning without violation of these annotations'
> > design concept. i.e especially cleaning up for @PublicEvolving APIs since
> > they are customer-facing. Regular cleaning up with all other
> @Experimental
> > and @Internal APIs would be even better, if there might be some APIs
> marked
> > as @deprecated.
> >
> > Best regards,
> > Jing
> >
> >
> >
> > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > One major issue we have, afaiu, is caused by the lack of
> > > housekeeping/house
> > > > cleaning, there are many APIs that were marked as deprecated a few
> > years
> > > > ago and still don't get removed. Some APIs should be easy to remove
> and
> > > > others will need some more clear rules, like the issue discussed at
> > [1].
> > >
> > This is by design. Most of these are @Public APIs that we had to carry
> > > around until Flink 2.0, because that was the initial guarantee that we
> > > gave people.
> >
> >
> > >
> > > As for the FLIP, I like the idea of explicitly writing down a
> > > deprecation period for APIs, particularly PublicEvolving ones.
> > > For Experimental I don't think it'd be a problem if we could change
> them
> > > right away,
> > > but looking back a bit I don't think it hurts us to also enforce some
> > > deprecation period.
> > > 1 release for both of these sound fine to me.
> > >
> > >
> > > My major point of contention is the removal of Public APIs between
> minor
> > > versions.
> > > This to me would a major setback towards a simpler upgrade path for
> > users.
> > > If these can be removed in minor versions than what even is a major
> > > release?
> > > The very definition we have for Public APIs is that they stick around
> > > until the next major one.
> > > Any rule that theoretically allows for breaking changes in Public API
> in
> > > every minor release is in my opinion not a viable option.
> > >
> > >
> > > The "carry recent Public APIs forward into the next major release"
> thing
> > > seems to presume a linear release history (aka, if 2.0 is released
> after
> > > 1.20, then there will be no 1.21), which I doubt will be the case. The
> > > idea behind it is good, but I'd say the right conclusion would be to
> not
> > > make that API public if we know a new major release hits in 3 months
> and
> > > is about to modify it. With a regular schedule for major releases this
> > > wouldn't be difficult to do.
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Becket,

Thanks for the clarification. Sorry that I didn't make myself clear enough.
Let me share more thoughts.


> Deprecating an API is just a more elegant way of replacing an API with a
> new one. The only difference between the two is whether the old API is kept
> and coexists with the new API for some releases or not. For end users,
> deprecation-then-remove is much more friendly than direct replacement.
>

Making old experimental API @deprecated with new API as replacement needs
extra effort. Let's use the example in FLIP to see what happened.

Deprecation rule A:

- API foo was introduced in 1.18.x
- API foo was carried over in 1.19.0
- API foo was marked as @deprecated in 1.19.1
- API foo is moved in 1.19.2 - the breaking change

Deprecation rule B:

- API foo was introduced in 1.18.x
- API foo was carried over in 1.19.0
- API foo was moved in 1.19.1 - the breaking change
- Another release 1.19.2

1.19.2 is the current/last release.

One key concern of users is to limit the effort as much as possible for any
patch upgrade. Let's name it "the rule of patch upgrade effort". If this is
not true, I'd like to know more different thoughts.

There will be following scenarios:

Scenarios 1: Following the rule of patch upgrade effort:

1. upgrade from 1.18.x to 1.19.x. This is an upgrade across minor releases.
Users will upgrade to 1.19.2(the last release), since there will be many
bigger changes than the foo-bar change. There is no difference between
deprecation rule A and B.
2. upgrade from 1.19.0 to 1.19.x. To minimize the upgrade effort, with the
deprecation rule A, users will upgrade to 1.19.1 while with the deprecation
rule B, users will stay with 1.19.0 and keep waiting for the next minor
release to handle breaking changes with effort. Since the last release is
1.19.2, users are using an old release in both cases. The deprecation rule
A is only slightly better, when there are other bug-fixes in 1.19.1 that
are urgently required for them. In this case, we will go to the second
scenario.

Scenarios 2: Don't follow the rule of patch upgrade effort:

1. upgrade from 1.18.x to 1.19.x. Same as in scenario 1. This is an upgrade
across minor releases. Users will upgrade to 1.19.2(the last release).
There is no difference between deprecation rule A and B.
2. upgrade from 1.19.0 to 1.19.x. Focus will be on upgrading to the last
version instead of effort. With both the deprecation rule A and B, users
will handle the breaking change and upgrade to 1.19.2. There is no
difference between them.

As we can see, with the (big) effort of maintaining old experimental APIs,
the value is very limited. And in scenario 1.2, since the new patch release
1.19.1 contains urgent bug-fixes, it is acceptable for those users who
really need them to put some effort into upgrading (Deprecation rule B).
After all, everyone was aware that @experimental can be changed between
patch releases in the first place.

This is orthogonal to the deprecation process, and may not even be required
> in some cases if the function changes by itself. But in general, this
> relies on the developer to decide. A simple test on readiness is to see if
> all the UT / IT cases written with the old API can be migrated to the new
> one and still work.  If the new API is not ready, we probably should not
> deprecate the old one to begin with.
>

I have a different thought. My gut feeling is that, once this FLIP is done,
developers will use it as the golden rule each time when deprecated APIs
need to be removed. For complex features, the deprecation phase will take
time, Marking old APIs deprecated as early as possible will give users more
time to migrate or give them the heads-up earlier. It is more comfortable
for users than having everything read and marking old APIs as deprecated
sharply and removing them after a defined migration period. Because the
longer we let such abandoned APIs not be marked as deprecated, the more
dependencies users will be building with them, and the bigger effort for
users to do migration in the future.



> By "backwards compatible", do you mean functionally equivalent? If the new
> APIs are designed to be not backwards compatible, then removing the
> deprecated source code is definitely allowed. If we don't think the new API
> is ready to take over the place for the old one, then we should wait. The
> migration period is the minimum time we have to wait before removing the
> source code. A longer migration period is OK.
>
>
Yes, by "backwards compatible", I meant functionally equivalent. You can
find more details in this thread[1]. I should have used "functionally
compatible" instead. Sorry for the confusion. This is also the example I
want to point out for the thought described above. If the FLIP does not
contain enough information, guidelines, conditions, the migration period
will be the only rule to decide when to remove deprecated APIs without
caring of anything else like functionality compatibility. If the migration
period described in this FLIP is only the minimum time, I think we still
have the house cleaning issue unsolved. Minimum means deprecated APIs can
not be removed before the migration period expires. The issue I was aware
of is when/how to remove APIs after the migration period expired, e.g.
PublicEvolving APIs that have been marked as deprecated years ago.


Best regards,
Jing

[1] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o

On Wed, Jun 14, 2023 at 9:56 AM Becket Qin <be...@gmail.com> wrote:

> Hi Jing,
>
> Thanks for the feedback. Please see the answers to your questions below:
>
> *"Always add a "Since X.X.X" comment to indicate when was a class /
> > interface / method marked as deprecated."*
> >  Could you describe it with a code example? Do you mean Java comments?
>
> It is just a comment such as "Since 1.18. Use X
> <
> https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)
> >XX
> instead.". And we can then look it up in the deprecated list[1] in each
> release and see which method should / can be deprecated.
>
> *"At least 1 patch release for the affected minor release for
> > Experimental APIs"*
> > The rule is absolutely right. However, afaiac, deprecation is different
> as
> > modification. As a user/dev, I would appreciate, if I do not need to do
> any
> > migration work for any deprecated API between patch releases upgrade.
> BTW,
> > if experimental APIs are allowed to change between patches, could we just
> > change them instead of marking them as deprecated and create new ones to
> > replace them?
>
> Deprecating an API is just a more elegant way of replacing an API with a
> new one. The only difference between the two is whether the old API is kept
> and coexists with the new API for some releases or not. For end users,
> deprecation-then-remove is much more friendly than direct replacement.
>
> 1. How to make sure the new APIs cover all functionality, i.e. backward
> > compatible, before removing the deprecated APIs? Since the
> > functionalities could only be built with the new APIs iteratively, there
> > will be a while (might be longer than the migration period) that the new
> > APIs are not backward compatible with the deprecated ones.
>
>

> This is orthogonal to the deprecation process, and may not even be required
> in some cases if the function changes by itself. But in general, this
> relies on the developer to decide. A simple test on readiness is to see if
> all the UT / IT cases written with the old API can be migrated to the new
> one and still work.  If the new API is not ready, we probably should not
> deprecate the old one to begin with.
>


>
> 2. Is it allowed to remove the deprecated APIs after the defined migration
> > period expires while the new APis are still not backward compatible?
>
>

> By "backwards compatible", do you mean functionally equivalent? If the new
> APIs are designed to be not backwards compatible, then removing the
> deprecated source code is definitely allowed. If we don't think the new API
> is ready to take over the place for the old one, then we should wait. The
> migration period is the minimum time we have to wait before removing the
> source code. A longer migration period is OK.
>
>

> 3. For the case of core API upgrade with downstream implementations, e.g.
> > connectors, What is the feasible deprecation strategy? Option1 bottom-up:
> > make sure the downstream implementation is backward compatible before
> > removing the deprecated core APIs. Option2 top-down: once the downstream
> > implementation of new APIs works fine, we can remove the deprecated core
> > APIs after the migration period expires. The implementation of the
> > deprecated APIs will not get any further update in upcoming releases(it
> has
> > been removed). There might be some missing features in the downstream
> > implementation of new APIs compared to the old implementation. Both
> options
> > have their own pros and cons.
>
> The downstream projects such as connectors in Flink should also follow the
> migration path we tell our users. i.e. If there is a cascading backwards
> incompatible change in the connectors due to a backwards incompatible
> change in the core, and as a result a longer migration period is required,
> then I think we should postpone the removal of source code. But in general,
> we should be able to provide the same migration period in the connectors as
> the flink core, if the connectors are upgraded to the latest version of
> core promptly.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html
>
>
> On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > > This is by design. Most of these are @Public APIs that we had to carry
> > > around until Flink 2.0, because that was the initial guarantee that we
> > > gave people.
> > >
> >
> > True, I knew @Public APIs could not be removed before the next major
> > release. I meant house cleaning without violation of these annotations'
> > design concept. i.e especially cleaning up for @PublicEvolving APIs since
> > they are customer-facing. Regular cleaning up with all other
> @Experimental
> > and @Internal APIs would be even better, if there might be some APIs
> marked
> > as @deprecated.
> >
> > Best regards,
> > Jing
> >
> >
> >
> > On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > On 13/06/2023 12:50, Jing Ge wrote:
> > > > One major issue we have, afaiu, is caused by the lack of
> > > housekeeping/house
> > > > cleaning, there are many APIs that were marked as deprecated a few
> > years
> > > > ago and still don't get removed. Some APIs should be easy to remove
> and
> > > > others will need some more clear rules, like the issue discussed at
> > [1].
> > >
> > This is by design. Most of these are @Public APIs that we had to carry
> > > around until Flink 2.0, because that was the initial guarantee that we
> > > gave people.
> >
> >
> > >
> > > As for the FLIP, I like the idea of explicitly writing down a
> > > deprecation period for APIs, particularly PublicEvolving ones.
> > > For Experimental I don't think it'd be a problem if we could change
> them
> > > right away,
> > > but looking back a bit I don't think it hurts us to also enforce some
> > > deprecation period.
> > > 1 release for both of these sound fine to me.
> > >
> > >
> > > My major point of contention is the removal of Public APIs between
> minor
> > > versions.
> > > This to me would a major setback towards a simpler upgrade path for
> > users.
> > > If these can be removed in minor versions than what even is a major
> > > release?
> > > The very definition we have for Public APIs is that they stick around
> > > until the next major one.
> > > Any rule that theoretically allows for breaking changes in Public API
> in
> > > every minor release is in my opinion not a viable option.
> > >
> > >
> > > The "carry recent Public APIs forward into the next major release"
> thing
> > > seems to presume a linear release history (aka, if 2.0 is released
> after
> > > 1.20, then there will be no 1.21), which I doubt will be the case. The
> > > idea behind it is good, but I'd say the right conclusion would be to
> not
> > > make that API public if we know a new major release hits in 3 months
> and
> > > is about to modify it. With a regular schedule for major releases this
> > > wouldn't be difficult to do.
> > >
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Jing,

Thanks for the feedback. Please see the answers to your questions below:

*"Always add a "Since X.X.X" comment to indicate when was a class /
> interface / method marked as deprecated."*
>  Could you describe it with a code example? Do you mean Java comments?

It is just a comment such as "Since 1.18. Use X
<https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/admin/Admin.html#incrementalAlterConfigs(java.util.Map)>XX
instead.". And we can then look it up in the deprecated list[1] in each
release and see which method should / can be deprecated.

*"At least 1 patch release for the affected minor release for
> Experimental APIs"*
> The rule is absolutely right. However, afaiac, deprecation is different as
> modification. As a user/dev, I would appreciate, if I do not need to do any
> migration work for any deprecated API between patch releases upgrade. BTW,
> if experimental APIs are allowed to change between patches, could we just
> change them instead of marking them as deprecated and create new ones to
> replace them?

Deprecating an API is just a more elegant way of replacing an API with a
new one. The only difference between the two is whether the old API is kept
and coexists with the new API for some releases or not. For end users,
deprecation-then-remove is much more friendly than direct replacement.

1. How to make sure the new APIs cover all functionality, i.e. backward
> compatible, before removing the deprecated APIs? Since the
> functionalities could only be built with the new APIs iteratively, there
> will be a while (might be longer than the migration period) that the new
> APIs are not backward compatible with the deprecated ones.

This is orthogonal to the deprecation process, and may not even be required
in some cases if the function changes by itself. But in general, this
relies on the developer to decide. A simple test on readiness is to see if
all the UT / IT cases written with the old API can be migrated to the new
one and still work.  If the new API is not ready, we probably should not
deprecate the old one to begin with.

2. Is it allowed to remove the deprecated APIs after the defined migration
> period expires while the new APis are still not backward compatible?

By "backwards compatible", do you mean functionally equivalent? If the new
APIs are designed to be not backwards compatible, then removing the
deprecated source code is definitely allowed. If we don't think the new API
is ready to take over the place for the old one, then we should wait. The
migration period is the minimum time we have to wait before removing the
source code. A longer migration period is OK.

3. For the case of core API upgrade with downstream implementations, e.g.
> connectors, What is the feasible deprecation strategy? Option1 bottom-up:
> make sure the downstream implementation is backward compatible before
> removing the deprecated core APIs. Option2 top-down: once the downstream
> implementation of new APIs works fine, we can remove the deprecated core
> APIs after the migration period expires. The implementation of the
> deprecated APIs will not get any further update in upcoming releases(it has
> been removed). There might be some missing features in the downstream
> implementation of new APIs compared to the old implementation. Both options
> have their own pros and cons.

The downstream projects such as connectors in Flink should also follow the
migration path we tell our users. i.e. If there is a cascading backwards
incompatible change in the connectors due to a backwards incompatible
change in the core, and as a result a longer migration period is required,
then I think we should postpone the removal of source code. But in general,
we should be able to provide the same migration period in the connectors as
the flink core, if the connectors are upgraded to the latest version of
core promptly.

Thanks,

Jiangjie (Becket) Qin

[1]
https://nightlies.apache.org/flink/flink-docs-master/api/java/deprecated-list.html


On Wed, Jun 14, 2023 at 1:15 AM Jing Ge <ji...@ververica.com.invalid> wrote:

> > This is by design. Most of these are @Public APIs that we had to carry
> > around until Flink 2.0, because that was the initial guarantee that we
> > gave people.
> >
>
> True, I knew @Public APIs could not be removed before the next major
> release. I meant house cleaning without violation of these annotations'
> design concept. i.e especially cleaning up for @PublicEvolving APIs since
> they are customer-facing. Regular cleaning up with all other @Experimental
> and @Internal APIs would be even better, if there might be some APIs marked
> as @deprecated.
>
> Best regards,
> Jing
>
>
>
> On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > On 13/06/2023 12:50, Jing Ge wrote:
> > > One major issue we have, afaiu, is caused by the lack of
> > housekeeping/house
> > > cleaning, there are many APIs that were marked as deprecated a few
> years
> > > ago and still don't get removed. Some APIs should be easy to remove and
> > > others will need some more clear rules, like the issue discussed at
> [1].
> >
> This is by design. Most of these are @Public APIs that we had to carry
> > around until Flink 2.0, because that was the initial guarantee that we
> > gave people.
>
>
> >
> > As for the FLIP, I like the idea of explicitly writing down a
> > deprecation period for APIs, particularly PublicEvolving ones.
> > For Experimental I don't think it'd be a problem if we could change them
> > right away,
> > but looking back a bit I don't think it hurts us to also enforce some
> > deprecation period.
> > 1 release for both of these sound fine to me.
> >
> >
> > My major point of contention is the removal of Public APIs between minor
> > versions.
> > This to me would a major setback towards a simpler upgrade path for
> users.
> > If these can be removed in minor versions than what even is a major
> > release?
> > The very definition we have for Public APIs is that they stick around
> > until the next major one.
> > Any rule that theoretically allows for breaking changes in Public API in
> > every minor release is in my opinion not a viable option.
> >
> >
> > The "carry recent Public APIs forward into the next major release" thing
> > seems to presume a linear release history (aka, if 2.0 is released after
> > 1.20, then there will be no 1.21), which I doubt will be the case. The
> > idea behind it is good, but I'd say the right conclusion would be to not
> > make that API public if we know a new major release hits in 3 months and
> > is about to modify it. With a regular schedule for major releases this
> > wouldn't be difficult to do.
> >
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
> This is by design. Most of these are @Public APIs that we had to carry
> around until Flink 2.0, because that was the initial guarantee that we
> gave people.
>

True, I knew @Public APIs could not be removed before the next major
release. I meant house cleaning without violation of these annotations'
design concept. i.e especially cleaning up for @PublicEvolving APIs since
they are customer-facing. Regular cleaning up with all other @Experimental
and @Internal APIs would be even better, if there might be some APIs marked
as @deprecated.

Best regards,
Jing



On Tue, Jun 13, 2023 at 4:25 PM Chesnay Schepler <ch...@apache.org> wrote:

> On 13/06/2023 12:50, Jing Ge wrote:
> > One major issue we have, afaiu, is caused by the lack of
> housekeeping/house
> > cleaning, there are many APIs that were marked as deprecated a few years
> > ago and still don't get removed. Some APIs should be easy to remove and
> > others will need some more clear rules, like the issue discussed at [1].
>
This is by design. Most of these are @Public APIs that we had to carry
> around until Flink 2.0, because that was the initial guarantee that we
> gave people.


>
> As for the FLIP, I like the idea of explicitly writing down a
> deprecation period for APIs, particularly PublicEvolving ones.
> For Experimental I don't think it'd be a problem if we could change them
> right away,
> but looking back a bit I don't think it hurts us to also enforce some
> deprecation period.
> 1 release for both of these sound fine to me.
>
>
> My major point of contention is the removal of Public APIs between minor
> versions.
> This to me would a major setback towards a simpler upgrade path for users.
> If these can be removed in minor versions than what even is a major
> release?
> The very definition we have for Public APIs is that they stick around
> until the next major one.
> Any rule that theoretically allows for breaking changes in Public API in
> every minor release is in my opinion not a viable option.
>
>
> The "carry recent Public APIs forward into the next major release" thing
> seems to presume a linear release history (aka, if 2.0 is released after
> 1.20, then there will be no 1.21), which I doubt will be the case. The
> idea behind it is good, but I'd say the right conclusion would be to not
> make that API public if we know a new major release hits in 3 months and
> is about to modify it. With a regular schedule for major releases this
> wouldn't be difficult to do.
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Chesnay Schepler <ch...@apache.org>.
On 13/06/2023 12:50, Jing Ge wrote:
> One major issue we have, afaiu, is caused by the lack of housekeeping/house
> cleaning, there are many APIs that were marked as deprecated a few years
> ago and still don't get removed. Some APIs should be easy to remove and
> others will need some more clear rules, like the issue discussed at [1].

This is by design. Most of these are @Public APIs that we had to carry 
around until Flink 2.0, because that was the initial guarantee that we 
gave people.


As for the FLIP, I like the idea of explicitly writing down a 
deprecation period for APIs, particularly PublicEvolving ones.
For Experimental I don't think it'd be a problem if we could change them 
right away,
but looking back a bit I don't think it hurts us to also enforce some 
deprecation period.
1 release for both of these sound fine to me.


My major point of contention is the removal of Public APIs between minor 
versions.
This to me would a major setback towards a simpler upgrade path for users.
If these can be removed in minor versions than what even is a major release?
The very definition we have for Public APIs is that they stick around 
until the next major one.
Any rule that theoretically allows for breaking changes in Public API in 
every minor release is in my opinion not a viable option.


The "carry recent Public APIs forward into the next major release" thing 
seems to presume a linear release history (aka, if 2.0 is released after 
1.20, then there will be no 1.21), which I doubt will be the case. The 
idea behind it is good, but I'd say the right conclusion would be to not 
make that API public if we know a new major release hits in 3 months and 
is about to modify it. With a regular schedule for major releases this 
wouldn't be difficult to do.

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Becket,

Thanks for driving this important topic! There were many discussions
previously that ended up with waiting up for a clear API deprecation
process definition. This FLIP will help a lot.

I'd like to ask some questions to understand your thoughts.

Speaking of the FLIP,

*"Always add a "Since X.X.X" comment to indicate when was a class /
interface / method marked as deprecated."*
 Could you describe it with a code example? Do you mean Java comments?

*"At least 1 patch release for the affected minor release for
Experimental APIs"*
The rule is absolutely right. However, afaiac, deprecation is different as
modification. As a user/dev, I would appreciate, if I do not need to do any
migration work for any deprecated API between patch releases upgrade. BTW,
if experimental APIs are allowed to change between patches, could we just
change them instead of marking them as deprecated and create new ones to
replace them?

One major issue we have, afaiu, is caused by the lack of housekeeping/house
cleaning, there are many APIs that were marked as deprecated a few years
ago and still don't get removed. Some APIs should be easy to remove and
others will need some more clear rules, like the issue discussed at [1].

Some common questions could be:

1. How to make sure the new APIs cover all functionality, i.e. backward
compatible, before removing the deprecated APIs? Since the
functionalities could only be built with the new APIs iteratively, there
will be a while (might be longer than the migration period) that the new
APIs are not backward compatible with the deprecated ones.
2. Is it allowed to remove the deprecated APIs after the defined migration
period expires while the new APis are still not backward compatible?
3. For the case of core API upgrade with downstream implementations, e.g.
connectors, What is the feasible deprecation strategy? Option1 bottom-up:
make sure the downstream implementation is backward compatible before
removing the deprecated core APIs. Option2 top-down: once the downstream
implementation of new APIs works fine, we can remove the deprecated core
APIs after the migration period expires. The implementation of the
deprecated APIs will not get any further update in upcoming releases(it has
been removed). There might be some missing features in the downstream
implementation of new APIs compared to the old implementation. Both options
have their own pros and cons.


Best regards,
Jing


[1] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o


On Mon, Jun 12, 2023 at 6:31 PM Stefan Richter
<sr...@confluent.io.invalid> wrote:

> Hi,
>
> Thanks a lot for bringing up this topic and for the initial proposal. As
> more and more people are looking into running Flink as a continuous service
> this discussion is becoming very relevant.
>
> What I would like to see is a clearer definition for what we understand by
> stability and compatibility. Our current policy only talks about being able
> to “compile” and “run” with a different version. As far as I can see, there
> is no guarantee about the stability of observable behavior. I believe it’s
> important for the community to include this important aspect in the
> guarantees that we give as our policy.
>
> For all changes that we do to the stable parts of the API, we should also
> consider how easy or difficult different types of changes would be for
> running Flink as a service with continuous delivery. For example,
> introducing a new interface to evolve the methods would make it easier to
> write adapter code than changing method signatures in-place on the existing
> interface. Those concerns should be considered in our process for evolving
> interfaces.
>
> Best,
> Stefan
>
>
>
>   <https://www.confluent.io/>
> Stefan Richter
> Principal Engineer II
>
> Follow us:  <
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>
> <https://twitter.com/ConfluentInc>
>
>
>
> > On 11. Jun 2023, at 14:30, Becket Qin <be...@gmail.com> wrote:
> >
> > Hi folks,
> >
> > As one of the release 2.0 efforts, the release managers were discussing
> our
> > API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
> > are relevant to this topic. These two FLIPs defined the stability
> guarantee
> > of the programming APIs with various different stability annotations, and
> > the promotion process. A recap of the conclusion is following:
> >
> > Stability:
> > @Internal API: can change between major/minor/patch releases.
> > @Experimental API: can change between major/minor/patch releases.
> > @PublicEvolving API: can change between major/minor releases.
> > @Public API: can only change between major releases.
> >
> > Promotion:
> > An @Experimental API should be promoted to @PublicEvolving after two
> > releases, and a @PublicEvolving API should be promoted to @Public API
> after
> > two releases, unless there is a documented reason not to do so.
> >
> > One thing not mentioned in these two FLIPs is the API deprecation
> process,
> > which is in fact critical and fundamental to how the stability guarantee
> is
> > provided in practice, because the stability is all about removing
> existing
> > APIs. For example, if we want to change a method "ResultFoo
> foo(ArgumentFoo
> > arg)" to "ResultBar bar(ArgumentBar arg)", there will be two ways to do
> > this:
> >
> > 1. Mark method "foo" as deprecated and add the new method "bar". At some
> > point later, remove the method "foo".
> > 2. Simply change the API in place, that basically means removing method
> foo
> > and adding method bar at the same time.
> >
> > In the first option, users are given a period with stability guarantee to
> > migrate from "foo" to "bar". For the second option, this migration period
> > is effectively 0. A zero migration period is problematic because end
> users
> > may need a feature/bug fix from a new version, but cannot upgrade right
> > away due to some backwards compatible changes, even though these changes
> > perfectly comply with the API stability guarantees defined above. So the
> > migration period is critical to the API stability guarantees for the end
> > users.
> >
> > The migration period is essentially how long a deprecated API can be
> > removed from the source code. So with this FLIP, I'd like to kick off the
> > discussion about our deprecation process.
> >
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%253A%2BIntroduce%2Ban%2BAPI%2Bdeprecation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0h_j72PiGBNM3BFmuHUOis
> >
> > Comments are welcome!
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > [1]
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%253A%2BSource%2BAPI%2Bstability%2Bguarantees&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0VDOsdIOFCOsGLlpPJq-ZD
> > [2]
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%253A%2BAPI%2Bstability%2Bgraduation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw11GfC5R4cD44a8amORO8EY
>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Stefan Richter <sr...@confluent.io.INVALID>.
Hi Becket,



> *1. What is the scope of public interfaces?*
> We actually already have a definition of public interfaces in the FLIP[1]
> wiki, it is basically anything that is sensible to the users, including
> packages, classes, method signature and behavior (blocking v.s.
> non-blocking, for example), metrics, configurations, CLI tools and
> arguments, and so on.

I agree with the scope that is defined by the annotation and think all observable behavior and also semantics should be covered by the policy. We should also review that all classes/methods/etc that are reachable through the API are annotated and define a sensible default in case something was missed. 

> 
> *2. What API changes can happen across different versions?*
> FLIP-196 answers the first question by defining in which versions
> programming APIs (methods and classes annotated with "Public",
> "PublicEvolving" or "Experimental") can have breaking changes. This allows
> us to get rid of deprecated APIs in patch / minor / major versions, while
> providing a clear expectation for our users.

I think the table has some contradiction with your first email for public API:

> @Public API: can only change between major releases.

Vs

> Public	At least 2 minor releases.


I assume the entry in the table is a mistake?

> 
> *3. When can breaking changes happen?*
> This FLIP tries to answer the second question, i.e. how users can adapt to
> breaking changes. We should avoid sudden API changes between two versions,
> and always leave time for the users to have a planned migration away from
> breaking changes. This basically means we will always take two steps when
> making breaking changes:
> 1. Introduce the new API (if needed), and mark the old API as deprecated.
> 2. After some time, which are the migration periods defined in this FLIP,
> remove the old API.
> 
> And in this FLIP, we would like to define the exact length of the migration
> period. It is going to be a trade-off between the maintenance cost of
> deprecated APIs and the time users have to migrate away from those
> deprecated APIs. In this FLIP we want to define the minimum lengths of the
> migration periods.
> 
> *4. How do users upgrade to Flink versions with breaking changes?*
> With the answers to the above three questions, the user upgrade path should
> be simple and clear:
> 1. upgrade to a Flink version that contains both the deprecated and new API.
> 2. have a planned migration to move to the new API.
> 3. upgrade to later Flink versions in which the code of the deprecated API
> is removed.


This is where things get interesting when we want to make Flink fit for running as a service. So far our release and compatibility model is very developer centric (easy for development to move forward quickly) and not user friendly. In SaaS, there is not necessarily a user available to upgrade a job when the service undergoes an upgrade. And the expectation of a user would be that a service can update without breaking their continuously running jobs. We should answer the question what level of stability we want to provide our user in the future and elevate the Flink technology to a level that allows to create a service with continuous delivery across versions. Other software, like e.g. the Linux OS gives way stronger stability guarantees across versions, and I wonder if the discussion should not also explore how we can reach a comparable level of stability for Flink.

> 
> So, it looks like our story for API stability and compatibility would be
> complete with this FLIP.
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> 
> On Tue, Jun 13, 2023 at 12:30 AM Stefan Richter
> <srichter@confluent.io.invalid <ma...@confluent.io.invalid>> wrote:
> 
>> Hi,
>> 
>> Thanks a lot for bringing up this topic and for the initial proposal. As
>> more and more people are looking into running Flink as a continuous service
>> this discussion is becoming very relevant.
>> 
>> What I would like to see is a clearer definition for what we understand by
>> stability and compatibility. Our current policy only talks about being able
>> to “compile” and “run” with a different version. As far as I can see, there
>> is no guarantee about the stability of observable behavior. I believe it’s
>> important for the community to include this important aspect in the
>> guarantees that we give as our policy.
>> 
>> For all changes that we do to the stable parts of the API, we should also
>> consider how easy or difficult different types of changes would be for
>> running Flink as a service with continuous delivery. For example,
>> introducing a new interface to evolve the methods would make it easier to
>> write adapter code than changing method signatures in-place on the existing
>> interface. Those concerns should be considered in our process for evolving
>> interfaces.
>> 
>> Best,
>> Stefan
>> 
>> 
>> 
>>  <https://www.google.com/url?q=https://www.confluent.io/&source=gmail-imap&ust=1687256610000000&usg=AOvVaw1dp7aAKHlg2BfLxN_FiCLh>
>> Stefan Richter
>> Principal Engineer II
>> 
>> Follow us:  <
>> https://www.google.com/url?q=https://www.confluent.io/blog?utm_source%3Dfooter%26utm_medium%3Demail%26utm_campaign%3Dch.email-signature_type.community_content.blog&source=gmail-imap&ust=1687256610000000&usg=AOvVaw1sXHw7whbn-edlT6E0L2XS>
>> <https://www.google.com/url?q=https://twitter.com/ConfluentInc&source=gmail-imap&ust=1687256610000000&usg=AOvVaw0HINEzFuW8kbEyJCdJMLui>
>> 
>> 
>> 
>>> On 11. Jun 2023, at 14:30, Becket Qin <becket.qin@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Hi folks,
>>> 
>>> As one of the release 2.0 efforts, the release managers were discussing
>> our
>>> API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
>>> are relevant to this topic. These two FLIPs defined the stability
>> guarantee
>>> of the programming APIs with various different stability annotations, and
>>> the promotion process. A recap of the conclusion is following:
>>> 
>>> Stability:
>>> @Internal API: can change between major/minor/patch releases.
>>> @Experimental API: can change between major/minor/patch releases.
>>> @PublicEvolving API: can change between major/minor releases.
>>> @Public API: can only change between major releases.
>>> 
>>> Promotion:
>>> An @Experimental API should be promoted to @PublicEvolving after two
>>> releases, and a @PublicEvolving API should be promoted to @Public API
>> after
>>> two releases, unless there is a documented reason not to do so.
>>> 
>>> One thing not mentioned in these two FLIPs is the API deprecation
>> process,
>>> which is in fact critical and fundamental to how the stability guarantee
>> is
>>> provided in practice, because the stability is all about removing
>> existing
>>> APIs. For example, if we want to change a method "ResultFoo
>> foo(ArgumentFoo
>>> arg)" to "ResultBar bar(ArgumentBar arg)", there will be two ways to do
>>> this:
>>> 
>>> 1. Mark method "foo" as deprecated and add the new method "bar". At some
>>> point later, remove the method "foo".
>>> 2. Simply change the API in place, that basically means removing method
>> foo
>>> and adding method bar at the same time.
>>> 
>>> In the first option, users are given a period with stability guarantee to
>>> migrate from "foo" to "bar". For the second option, this migration period
>>> is effectively 0. A zero migration period is problematic because end
>> users
>>> may need a feature/bug fix from a new version, but cannot upgrade right
>>> away due to some backwards compatible changes, even though these changes
>>> perfectly comply with the API stability guarantees defined above. So the
>>> migration period is critical to the API stability guarantees for the end
>>> users.
>>> 
>>> The migration period is essentially how long a deprecated API can be
>>> removed from the source code. So with this FLIP, I'd like to kick off the
>>> discussion about our deprecation process.
>>> 
>>> 
>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/FLINK/FLIP-321%25253A%252BIntroduce%252Ban%252BAPI%252Bdeprecation%252Bprocess%26source%3Dgmail-imap%26ust%3D1687091523000000%26usg%3DAOvVaw0h_j72PiGBNM3BFmuHUOis&source=gmail-imap&ust=1687256610000000&usg=AOvVaw1Jm26zvqhwzD19JyokqEkR
>>> 
>>> Comments are welcome!
>>> 
>>> Thanks,
>>> 
>>> Jiangjie (Becket) Qin
>>> 
>>> [1]
>>> 
>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/FLINK/FLIP-196%25253A%252BSource%252BAPI%252Bstability%252Bguarantees%26source%3Dgmail-imap%26ust%3D1687091523000000%26usg%3DAOvVaw0VDOsdIOFCOsGLlpPJq-ZD&source=gmail-imap&ust=1687256610000000&usg=AOvVaw0VAAYlNPEe5g6s7Ob_bX7_
>>> [2]
>>> 
>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/FLINK/FLIP-197%25253A%252BAPI%252Bstability%252Bgraduation%252Bprocess%26source%3Dgmail-imap%26ust%3D1687091523000000%26usg%3DAOvVaw11GfC5R4cD44a8amORO8EY&source=gmail-imap&ust=1687256610000000&usg=AOvVaw1tBEpXV592z3on5y5X0sXD


Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Becket Qin <be...@gmail.com>.
Hi Stefan,

Thanks for the comments. I agree the API stability and compatibility
guarantees should be clear.
I think we need to answer the following four questions in order to complete
the the public API stability and compatibility story:

*1. What is the scope of public interfaces?*
We actually already have a definition of public interfaces in the FLIP[1]
wiki, it is basically anything that is sensible to the users, including
packages, classes, method signature and behavior (blocking v.s.
non-blocking, for example), metrics, configurations, CLI tools and
arguments, and so on.

*2. What API changes can happen across different versions?*
FLIP-196 answers the first question by defining in which versions
programming APIs (methods and classes annotated with "Public",
"PublicEvolving" or "Experimental") can have breaking changes. This allows
us to get rid of deprecated APIs in patch / minor / major versions, while
providing a clear expectation for our users.

*3. When can breaking changes happen?*
This FLIP tries to answer the second question, i.e. how users can adapt to
breaking changes. We should avoid sudden API changes between two versions,
and always leave time for the users to have a planned migration away from
breaking changes. This basically means we will always take two steps when
making breaking changes:
1. Introduce the new API (if needed), and mark the old API as deprecated.
2. After some time, which are the migration periods defined in this FLIP,
remove the old API.

And in this FLIP, we would like to define the exact length of the migration
period. It is going to be a trade-off between the maintenance cost of
deprecated APIs and the time users have to migrate away from those
deprecated APIs. In this FLIP we want to define the minimum lengths of the
migration periods.

*4. How do users upgrade to Flink versions with breaking changes?*
With the answers to the above three questions, the user upgrade path should
be simple and clear:
1. upgrade to a Flink version that contains both the deprecated and new API.
2. have a planned migration to move to the new API.
3. upgrade to later Flink versions in which the code of the deprecated API
is removed.

So, it looks like our story for API stability and compatibility would be
complete with this FLIP.

Thanks,

Jiangjie (Becket) Qin


On Tue, Jun 13, 2023 at 12:30 AM Stefan Richter
<sr...@confluent.io.invalid> wrote:

> Hi,
>
> Thanks a lot for bringing up this topic and for the initial proposal. As
> more and more people are looking into running Flink as a continuous service
> this discussion is becoming very relevant.
>
> What I would like to see is a clearer definition for what we understand by
> stability and compatibility. Our current policy only talks about being able
> to “compile” and “run” with a different version. As far as I can see, there
> is no guarantee about the stability of observable behavior. I believe it’s
> important for the community to include this important aspect in the
> guarantees that we give as our policy.
>
> For all changes that we do to the stable parts of the API, we should also
> consider how easy or difficult different types of changes would be for
> running Flink as a service with continuous delivery. For example,
> introducing a new interface to evolve the methods would make it easier to
> write adapter code than changing method signatures in-place on the existing
> interface. Those concerns should be considered in our process for evolving
> interfaces.
>
> Best,
> Stefan
>
>
>
>   <https://www.confluent.io/>
> Stefan Richter
> Principal Engineer II
>
> Follow us:  <
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>
> <https://twitter.com/ConfluentInc>
>
>
>
> > On 11. Jun 2023, at 14:30, Becket Qin <be...@gmail.com> wrote:
> >
> > Hi folks,
> >
> > As one of the release 2.0 efforts, the release managers were discussing
> our
> > API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
> > are relevant to this topic. These two FLIPs defined the stability
> guarantee
> > of the programming APIs with various different stability annotations, and
> > the promotion process. A recap of the conclusion is following:
> >
> > Stability:
> > @Internal API: can change between major/minor/patch releases.
> > @Experimental API: can change between major/minor/patch releases.
> > @PublicEvolving API: can change between major/minor releases.
> > @Public API: can only change between major releases.
> >
> > Promotion:
> > An @Experimental API should be promoted to @PublicEvolving after two
> > releases, and a @PublicEvolving API should be promoted to @Public API
> after
> > two releases, unless there is a documented reason not to do so.
> >
> > One thing not mentioned in these two FLIPs is the API deprecation
> process,
> > which is in fact critical and fundamental to how the stability guarantee
> is
> > provided in practice, because the stability is all about removing
> existing
> > APIs. For example, if we want to change a method "ResultFoo
> foo(ArgumentFoo
> > arg)" to "ResultBar bar(ArgumentBar arg)", there will be two ways to do
> > this:
> >
> > 1. Mark method "foo" as deprecated and add the new method "bar". At some
> > point later, remove the method "foo".
> > 2. Simply change the API in place, that basically means removing method
> foo
> > and adding method bar at the same time.
> >
> > In the first option, users are given a period with stability guarantee to
> > migrate from "foo" to "bar". For the second option, this migration period
> > is effectively 0. A zero migration period is problematic because end
> users
> > may need a feature/bug fix from a new version, but cannot upgrade right
> > away due to some backwards compatible changes, even though these changes
> > perfectly comply with the API stability guarantees defined above. So the
> > migration period is critical to the API stability guarantees for the end
> > users.
> >
> > The migration period is essentially how long a deprecated API can be
> > removed from the source code. So with this FLIP, I'd like to kick off the
> > discussion about our deprecation process.
> >
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%253A%2BIntroduce%2Ban%2BAPI%2Bdeprecation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0h_j72PiGBNM3BFmuHUOis
> >
> > Comments are welcome!
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > [1]
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%253A%2BSource%2BAPI%2Bstability%2Bguarantees&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0VDOsdIOFCOsGLlpPJq-ZD
> > [2]
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%253A%2BAPI%2Bstability%2Bgraduation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw11GfC5R4cD44a8amORO8EY
>
>

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Posted by Stefan Richter <sr...@confluent.io.INVALID>.
Hi,

Thanks a lot for bringing up this topic and for the initial proposal. As more and more people are looking into running Flink as a continuous service this discussion is becoming very relevant.

What I would like to see is a clearer definition for what we understand by stability and compatibility. Our current policy only talks about being able to “compile” and “run” with a different version. As far as I can see, there is no guarantee about the stability of observable behavior. I believe it’s important for the community to include this important aspect in the guarantees that we give as our policy.

For all changes that we do to the stable parts of the API, we should also consider how easy or difficult different types of changes would be for running Flink as a service with continuous delivery. For example, introducing a new interface to evolve the methods would make it easier to write adapter code than changing method signatures in-place on the existing interface. Those concerns should be considered in our process for evolving interfaces.

Best,
Stefan



  <https://www.confluent.io/>	
Stefan Richter
Principal Engineer II

Follow us:  <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog> <https://twitter.com/ConfluentInc>



> On 11. Jun 2023, at 14:30, Becket Qin <be...@gmail.com> wrote:
> 
> Hi folks,
> 
> As one of the release 2.0 efforts, the release managers were discussing our
> API lifecycle policies. There have been FLIP-196[1] and FLIP-197[2] that
> are relevant to this topic. These two FLIPs defined the stability guarantee
> of the programming APIs with various different stability annotations, and
> the promotion process. A recap of the conclusion is following:
> 
> Stability:
> @Internal API: can change between major/minor/patch releases.
> @Experimental API: can change between major/minor/patch releases.
> @PublicEvolving API: can change between major/minor releases.
> @Public API: can only change between major releases.
> 
> Promotion:
> An @Experimental API should be promoted to @PublicEvolving after two
> releases, and a @PublicEvolving API should be promoted to @Public API after
> two releases, unless there is a documented reason not to do so.
> 
> One thing not mentioned in these two FLIPs is the API deprecation process,
> which is in fact critical and fundamental to how the stability guarantee is
> provided in practice, because the stability is all about removing existing
> APIs. For example, if we want to change a method "ResultFoo foo(ArgumentFoo
> arg)" to "ResultBar bar(ArgumentBar arg)", there will be two ways to do
> this:
> 
> 1. Mark method "foo" as deprecated and add the new method "bar". At some
> point later, remove the method "foo".
> 2. Simply change the API in place, that basically means removing method foo
> and adding method bar at the same time.
> 
> In the first option, users are given a period with stability guarantee to
> migrate from "foo" to "bar". For the second option, this migration period
> is effectively 0. A zero migration period is problematic because end users
> may need a feature/bug fix from a new version, but cannot upgrade right
> away due to some backwards compatible changes, even though these changes
> perfectly comply with the API stability guarantees defined above. So the
> migration period is critical to the API stability guarantees for the end
> users.
> 
> The migration period is essentially how long a deprecated API can be
> removed from the source code. So with this FLIP, I'd like to kick off the
> discussion about our deprecation process.
> 
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%253A%2BIntroduce%2Ban%2BAPI%2Bdeprecation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0h_j72PiGBNM3BFmuHUOis
> 
> Comments are welcome!
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> [1]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%253A%2BSource%2BAPI%2Bstability%2Bguarantees&source=gmail-imap&ust=1687091523000000&usg=AOvVaw0VDOsdIOFCOsGLlpPJq-ZD
> [2]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%253A%2BAPI%2Bstability%2Bgraduation%2Bprocess&source=gmail-imap&ust=1687091523000000&usg=AOvVaw11GfC5R4cD44a8amORO8EY