You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2020/06/23 13:32:40 UTC
[DISCUSS] Cross-version compatibility guarantees of Flink
Modules/Jars
Hi,
this has come up a few times now and I think we need to discuss the
guarantees that we want to officially give for this. What I mean by
cross-version compatibility is using, say, a Flink 1.10 Kafka connector
dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
1.10.1. In the past, this hast mostly worked. I think this was largely
by accident, though.
The problem is that connectors, which might be annotated as @Public can
use classes internally which are annotated as @Internal or
@PublicEvolving. If those internal dependencies change, then the
connector jar can not be used for different versions. This has happened
at least in [1] and [2], where [2] was caused by the interplay of [3]
and [4]. The initial release note on [4] was saying that the Kafka 0.9
connector can be used with Flink 1.11, but this was rendered wrong by
[3]. (Also, sorry for all the []s ...)
What should we do about it? So far our strategy for ensuring that
jars/dependencies are compatible between versions was "hope". If we want
to really verify this compatibility we would have to ensure that
"public" code does not transitively use any "non-public" dependencies.
An alternative would be to say we don't support any cross-version
compatibility between Flink versions. If users want to use an older
connector they would have to copy the code, make sure it compiles
against a newer Flink version and then manage that themselves
What do you think?
Best,
Aljoscha
[1] https://issues.apache.org/jira/browse/FLINK-13586
[2] https://github.com/apache/flink/pull/12699#discussion_r442013514
[3] https://issues.apache.org/jira/browse/FLINK-17376
[4] https://issues.apache.org/jira/browse/FLINK-15115
Re: [DISCUSS] Cross-version compatibility guarantees of Flink
Modules/Jars
Posted by Chesnay Schepler <ch...@apache.org>.
> /I didn't check manually but I would say this probably affects all
connectors and we shouldn't recommend people use them across different
(minor, that is 1.x) versions. For patch versions it should be fine with
our recent discussion of not breaking @PublicEvolving for patch (1.x.y)
releases.
/
/
/
Well yes, but this does effectively mean that all user-jars should be
rebuilt against the new Flink version.
I don't think there even is a connector that is /really/ @Public, and
there won't be many jobs getting away with not using any connector.
//
On 24/06/2020 10:38, Aljoscha Krettek wrote:
> I recently did change the attitude there, see for example the new
> WatermarkStrategy and WatermarkGenerator and code around that. [1] [2]
>
> I think this is up to the judgement of folks that add new APIs,
> sometimes we don't know very well how things will shake out but I
> agree with Chesnay that we have been to liberal with @PublicEvolving.
>
> Coming back to cross-version compatibility for jars, I think all code
> that uses @PublicEvolving API is not guaranteed to be compatible
> across versions. I didn't check manually but I would say this probably
> affects all connectors and we shouldn't recommend people use them
> across different (minor, that is 1.x) versions. For patch versions it
> should be fine with our recent discussion of not breaking
> @PublicEvolving for patch (1.x.y) releases.
>
> Aljoscha
>
> [1]
> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java
> [2]
> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkGenerator.java
>
> On 24.06.20 09:39, Chesnay Schepler wrote:
>> Given that there are no compatibility guarantees for @PublicEvolving
>> classes I'm not sure how much value there is in basing this
>> discussion on their current state.
>>
>> Here's the thing; if we want connectors to be compatible across
>> versions, then this should also work for user-defined connectors
>> (e.g., distributed via flink-packages.org).
>> Because if this is not the case, it just reinforces the need of
>> having all connectors be maintained by us.
>>
>> This implies that compatibility testing _our connectors_ is not
>> sufficient,
>> and will require us to either redefine our guarantees for
>> @PublicEvolving classes, or define a @Public API for connectors to use.
>>
>> And this doesn't just apply to connectors, but also other APIs.
>> Take the ProcessFunction for example, that we recommend people to use
>> in the docs and mailing lists. No guarantees whatsoever.
>>
>> We have to rethink our compatibility story, because it just isn't sound.
>> Yes, we got burned in the past by having too much be @Public and
>> forcing us to keep things around for longer than we wanted,
>> but then we did a full 180 and stopped giving _any_ guarantees for
>> all new APIs.
>>
>> It is about time we change this.
>>
>> On 24/06/2020 09:15, Aljoscha Krettek wrote:
>>> Yes, I would say the situation is different for minor vs. patch.
>>>
>>> Side note: in a version like 1.x.y most people in the Flink
>>> community see x as the major version and y as the minor version. I
>>> know that this is not proper semver and I could be wrong about how
>>> people see it. It doesn't change anything about the discussion, though.
>>>
>>> Another side note: I said earlier that we would have to check the
>>> whole transitive closure for usage of "internal" classes. This is
>>> wrong: we only need to check the first level, i.e. code that is
>>> directly used by a connector (or other jar that we want to use
>>> across versions). This is still something that we don't do, so the
>>> fact that it works can be somewhat attributed to luck and the right
>>> people looking at the right thing at the right time.
>>>
>>> Aljoscha
>>>
>>> On 23.06.20 17:25, Konstantin Knauf wrote:
>>>> Hi Aljoscha,
>>>>
>>>> Thank you for bringing this up. IMHO the situation is different for
>>>> minor &
>>>> patch version upgrades.
>>>>
>>>> 1) I don't think we need to provide any guarantees across Flink minor
>>>> versions (e.g. 1.10.1 -> 1.11.0). It seems reasonable to expect
>>>> users to
>>>> recompile their user JARs when upgrading the cluster to the next minor
>>>> version of Apache Flink.
>>>>
>>>> 2) We should aim for compatibility across patch versions (1.10.0 ->
>>>> 1.10.1). Being able to apply a patch to the Flink runtime/framework
>>>> without
>>>> updating the dependencies of each user jar running on the cluster
>>>> seems
>>>> really useful to me.
>>>>
>>>> According to the discussion in [1] around stability guarantees for
>>>> @PublicEvolving, this would mean that connectors can use @Public
>>>> and @PublicEvolving classes, but not @Internal classes, right? This
>>>> generally seems reasonable to me. If we want to make it easy for
>>>> our users,
>>>> we need to aim for stable interfaces anyway.
>>>>
>>>> Cheers,
>>>>
>>>> Konstantin
>>>>
>>>> []
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stability-guarantees-for-PublicEvolving-classes-tp41459.html
>>>>
>>>>
>>>> On Tue, Jun 23, 2020 at 3:32 PM Aljoscha Krettek <al...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> this has come up a few times now and I think we need to discuss the
>>>>> guarantees that we want to officially give for this. What I mean by
>>>>> cross-version compatibility is using, say, a Flink 1.10 Kafka
>>>>> connector
>>>>> dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with
>>>>> Flink
>>>>> 1.10.1. In the past, this hast mostly worked. I think this was
>>>>> largely
>>>>> by accident, though.
>>>>>
>>>>> The problem is that connectors, which might be annotated as
>>>>> @Public can
>>>>> use classes internally which are annotated as @Internal or
>>>>> @PublicEvolving. If those internal dependencies change, then the
>>>>> connector jar can not be used for different versions. This has
>>>>> happened
>>>>> at least in [1] and [2], where [2] was caused by the interplay of [3]
>>>>> and [4]. The initial release note on [4] was saying that the Kafka
>>>>> 0.9
>>>>> connector can be used with Flink 1.11, but this was rendered wrong by
>>>>> [3]. (Also, sorry for all the []s ...)
>>>>>
>>>>> What should we do about it? So far our strategy for ensuring that
>>>>> jars/dependencies are compatible between versions was "hope". If
>>>>> we want
>>>>> to really verify this compatibility we would have to ensure that
>>>>> "public" code does not transitively use any "non-public"
>>>>> dependencies.
>>>>>
>>>>> An alternative would be to say we don't support any cross-version
>>>>> compatibility between Flink versions. If users want to use an older
>>>>> connector they would have to copy the code, make sure it compiles
>>>>> against a newer Flink version and then manage that themselves
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Best,
>>>>> Aljoscha
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-13586
>>>>> [2] https://github.com/apache/flink/pull/12699#discussion_r442013514
>>>>> [3] https://issues.apache.org/jira/browse/FLINK-17376
>>>>> [4] https://issues.apache.org/jira/browse/FLINK-15115
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
Re: [DISCUSS] Cross-version compatibility guarantees of Flink
Modules/Jars
Posted by Aljoscha Krettek <al...@apache.org>.
I recently did change the attitude there, see for example the new
WatermarkStrategy and WatermarkGenerator and code around that. [1] [2]
I think this is up to the judgement of folks that add new APIs,
sometimes we don't know very well how things will shake out but I agree
with Chesnay that we have been to liberal with @PublicEvolving.
Coming back to cross-version compatibility for jars, I think all code
that uses @PublicEvolving API is not guaranteed to be compatible across
versions. I didn't check manually but I would say this probably affects
all connectors and we shouldn't recommend people use them across
different (minor, that is 1.x) versions. For patch versions it should be
fine with our recent discussion of not breaking @PublicEvolving for
patch (1.x.y) releases.
Aljoscha
[1]
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java
[2]
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkGenerator.java
On 24.06.20 09:39, Chesnay Schepler wrote:
> Given that there are no compatibility guarantees for @PublicEvolving
> classes I'm not sure how much value there is in basing this discussion
> on their current state.
>
> Here's the thing; if we want connectors to be compatible across
> versions, then this should also work for user-defined connectors (e.g.,
> distributed via flink-packages.org).
> Because if this is not the case, it just reinforces the need of having
> all connectors be maintained by us.
>
> This implies that compatibility testing _our connectors_ is not sufficient,
> and will require us to either redefine our guarantees for
> @PublicEvolving classes, or define a @Public API for connectors to use.
>
> And this doesn't just apply to connectors, but also other APIs.
> Take the ProcessFunction for example, that we recommend people to use in
> the docs and mailing lists. No guarantees whatsoever.
>
> We have to rethink our compatibility story, because it just isn't sound.
> Yes, we got burned in the past by having too much be @Public and forcing
> us to keep things around for longer than we wanted,
> but then we did a full 180 and stopped giving _any_ guarantees for all
> new APIs.
>
> It is about time we change this.
>
> On 24/06/2020 09:15, Aljoscha Krettek wrote:
>> Yes, I would say the situation is different for minor vs. patch.
>>
>> Side note: in a version like 1.x.y most people in the Flink community
>> see x as the major version and y as the minor version. I know that
>> this is not proper semver and I could be wrong about how people see
>> it. It doesn't change anything about the discussion, though.
>>
>> Another side note: I said earlier that we would have to check the
>> whole transitive closure for usage of "internal" classes. This is
>> wrong: we only need to check the first level, i.e. code that is
>> directly used by a connector (or other jar that we want to use across
>> versions). This is still something that we don't do, so the fact that
>> it works can be somewhat attributed to luck and the right people
>> looking at the right thing at the right time.
>>
>> Aljoscha
>>
>> On 23.06.20 17:25, Konstantin Knauf wrote:
>>> Hi Aljoscha,
>>>
>>> Thank you for bringing this up. IMHO the situation is different for
>>> minor &
>>> patch version upgrades.
>>>
>>> 1) I don't think we need to provide any guarantees across Flink minor
>>> versions (e.g. 1.10.1 -> 1.11.0). It seems reasonable to expect users to
>>> recompile their user JARs when upgrading the cluster to the next minor
>>> version of Apache Flink.
>>>
>>> 2) We should aim for compatibility across patch versions (1.10.0 ->
>>> 1.10.1). Being able to apply a patch to the Flink runtime/framework
>>> without
>>> updating the dependencies of each user jar running on the cluster seems
>>> really useful to me.
>>>
>>> According to the discussion in [1] around stability guarantees for
>>> @PublicEvolving, this would mean that connectors can use @Public
>>> and @PublicEvolving classes, but not @Internal classes, right? This
>>> generally seems reasonable to me. If we want to make it easy for our
>>> users,
>>> we need to aim for stable interfaces anyway.
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> []
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stability-guarantees-for-PublicEvolving-classes-tp41459.html
>>>
>>>
>>> On Tue, Jun 23, 2020 at 3:32 PM Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> this has come up a few times now and I think we need to discuss the
>>>> guarantees that we want to officially give for this. What I mean by
>>>> cross-version compatibility is using, say, a Flink 1.10 Kafka connector
>>>> dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
>>>> 1.10.1. In the past, this hast mostly worked. I think this was largely
>>>> by accident, though.
>>>>
>>>> The problem is that connectors, which might be annotated as @Public can
>>>> use classes internally which are annotated as @Internal or
>>>> @PublicEvolving. If those internal dependencies change, then the
>>>> connector jar can not be used for different versions. This has happened
>>>> at least in [1] and [2], where [2] was caused by the interplay of [3]
>>>> and [4]. The initial release note on [4] was saying that the Kafka 0.9
>>>> connector can be used with Flink 1.11, but this was rendered wrong by
>>>> [3]. (Also, sorry for all the []s ...)
>>>>
>>>> What should we do about it? So far our strategy for ensuring that
>>>> jars/dependencies are compatible between versions was "hope". If we
>>>> want
>>>> to really verify this compatibility we would have to ensure that
>>>> "public" code does not transitively use any "non-public" dependencies.
>>>>
>>>> An alternative would be to say we don't support any cross-version
>>>> compatibility between Flink versions. If users want to use an older
>>>> connector they would have to copy the code, make sure it compiles
>>>> against a newer Flink version and then manage that themselves
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-13586
>>>> [2] https://github.com/apache/flink/pull/12699#discussion_r442013514
>>>> [3] https://issues.apache.org/jira/browse/FLINK-17376
>>>> [4] https://issues.apache.org/jira/browse/FLINK-15115
>>>>
>>>
>>>
>>
>>
>
Re: [DISCUSS] Cross-version compatibility guarantees of Flink
Modules/Jars
Posted by Chesnay Schepler <ch...@apache.org>.
Given that there are no compatibility guarantees for @PublicEvolving
classes I'm not sure how much value there is in basing this discussion
on their current state.
Here's the thing; if we want connectors to be compatible across
versions, then this should also work for user-defined connectors (e.g.,
distributed via flink-packages.org).
Because if this is not the case, it just reinforces the need of having
all connectors be maintained by us.
This implies that compatibility testing _our connectors_ is not sufficient,
and will require us to either redefine our guarantees for
@PublicEvolving classes, or define a @Public API for connectors to use.
And this doesn't just apply to connectors, but also other APIs.
Take the ProcessFunction for example, that we recommend people to use in
the docs and mailing lists. No guarantees whatsoever.
We have to rethink our compatibility story, because it just isn't sound.
Yes, we got burned in the past by having too much be @Public and forcing
us to keep things around for longer than we wanted,
but then we did a full 180 and stopped giving _any_ guarantees for all
new APIs.
It is about time we change this.
On 24/06/2020 09:15, Aljoscha Krettek wrote:
> Yes, I would say the situation is different for minor vs. patch.
>
> Side note: in a version like 1.x.y most people in the Flink community
> see x as the major version and y as the minor version. I know that
> this is not proper semver and I could be wrong about how people see
> it. It doesn't change anything about the discussion, though.
>
> Another side note: I said earlier that we would have to check the
> whole transitive closure for usage of "internal" classes. This is
> wrong: we only need to check the first level, i.e. code that is
> directly used by a connector (or other jar that we want to use across
> versions). This is still something that we don't do, so the fact that
> it works can be somewhat attributed to luck and the right people
> looking at the right thing at the right time.
>
> Aljoscha
>
> On 23.06.20 17:25, Konstantin Knauf wrote:
>> Hi Aljoscha,
>>
>> Thank you for bringing this up. IMHO the situation is different for
>> minor &
>> patch version upgrades.
>>
>> 1) I don't think we need to provide any guarantees across Flink minor
>> versions (e.g. 1.10.1 -> 1.11.0). It seems reasonable to expect users to
>> recompile their user JARs when upgrading the cluster to the next minor
>> version of Apache Flink.
>>
>> 2) We should aim for compatibility across patch versions (1.10.0 ->
>> 1.10.1). Being able to apply a patch to the Flink runtime/framework
>> without
>> updating the dependencies of each user jar running on the cluster seems
>> really useful to me.
>>
>> According to the discussion in [1] around stability guarantees for
>> @PublicEvolving, this would mean that connectors can use @Public
>> and @PublicEvolving classes, but not @Internal classes, right? This
>> generally seems reasonable to me. If we want to make it easy for our
>> users,
>> we need to aim for stable interfaces anyway.
>>
>> Cheers,
>>
>> Konstantin
>>
>> []
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stability-guarantees-for-PublicEvolving-classes-tp41459.html
>>
>>
>> On Tue, Jun 23, 2020 at 3:32 PM Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> this has come up a few times now and I think we need to discuss the
>>> guarantees that we want to officially give for this. What I mean by
>>> cross-version compatibility is using, say, a Flink 1.10 Kafka connector
>>> dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
>>> 1.10.1. In the past, this hast mostly worked. I think this was largely
>>> by accident, though.
>>>
>>> The problem is that connectors, which might be annotated as @Public can
>>> use classes internally which are annotated as @Internal or
>>> @PublicEvolving. If those internal dependencies change, then the
>>> connector jar can not be used for different versions. This has happened
>>> at least in [1] and [2], where [2] was caused by the interplay of [3]
>>> and [4]. The initial release note on [4] was saying that the Kafka 0.9
>>> connector can be used with Flink 1.11, but this was rendered wrong by
>>> [3]. (Also, sorry for all the []s ...)
>>>
>>> What should we do about it? So far our strategy for ensuring that
>>> jars/dependencies are compatible between versions was "hope". If we
>>> want
>>> to really verify this compatibility we would have to ensure that
>>> "public" code does not transitively use any "non-public" dependencies.
>>>
>>> An alternative would be to say we don't support any cross-version
>>> compatibility between Flink versions. If users want to use an older
>>> connector they would have to copy the code, make sure it compiles
>>> against a newer Flink version and then manage that themselves
>>>
>>> What do you think?
>>>
>>> Best,
>>> Aljoscha
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-13586
>>> [2] https://github.com/apache/flink/pull/12699#discussion_r442013514
>>> [3] https://issues.apache.org/jira/browse/FLINK-17376
>>> [4] https://issues.apache.org/jira/browse/FLINK-15115
>>>
>>
>>
>
>
Re: [DISCUSS] Cross-version compatibility guarantees of Flink
Modules/Jars
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, I would say the situation is different for minor vs. patch.
Side note: in a version like 1.x.y most people in the Flink community
see x as the major version and y as the minor version. I know that this
is not proper semver and I could be wrong about how people see it. It
doesn't change anything about the discussion, though.
Another side note: I said earlier that we would have to check the whole
transitive closure for usage of "internal" classes. This is wrong: we
only need to check the first level, i.e. code that is directly used by a
connector (or other jar that we want to use across versions). This is
still something that we don't do, so the fact that it works can be
somewhat attributed to luck and the right people looking at the right
thing at the right time.
Aljoscha
On 23.06.20 17:25, Konstantin Knauf wrote:
> Hi Aljoscha,
>
> Thank you for bringing this up. IMHO the situation is different for minor &
> patch version upgrades.
>
> 1) I don't think we need to provide any guarantees across Flink minor
> versions (e.g. 1.10.1 -> 1.11.0). It seems reasonable to expect users to
> recompile their user JARs when upgrading the cluster to the next minor
> version of Apache Flink.
>
> 2) We should aim for compatibility across patch versions (1.10.0 ->
> 1.10.1). Being able to apply a patch to the Flink runtime/framework without
> updating the dependencies of each user jar running on the cluster seems
> really useful to me.
>
> According to the discussion in [1] around stability guarantees for
> @PublicEvolving, this would mean that connectors can use @Public
> and @PublicEvolving classes, but not @Internal classes, right? This
> generally seems reasonable to me. If we want to make it easy for our users,
> we need to aim for stable interfaces anyway.
>
> Cheers,
>
> Konstantin
>
> []
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stability-guarantees-for-PublicEvolving-classes-tp41459.html
>
> On Tue, Jun 23, 2020 at 3:32 PM Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>>
>> this has come up a few times now and I think we need to discuss the
>> guarantees that we want to officially give for this. What I mean by
>> cross-version compatibility is using, say, a Flink 1.10 Kafka connector
>> dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
>> 1.10.1. In the past, this hast mostly worked. I think this was largely
>> by accident, though.
>>
>> The problem is that connectors, which might be annotated as @Public can
>> use classes internally which are annotated as @Internal or
>> @PublicEvolving. If those internal dependencies change, then the
>> connector jar can not be used for different versions. This has happened
>> at least in [1] and [2], where [2] was caused by the interplay of [3]
>> and [4]. The initial release note on [4] was saying that the Kafka 0.9
>> connector can be used with Flink 1.11, but this was rendered wrong by
>> [3]. (Also, sorry for all the []s ...)
>>
>> What should we do about it? So far our strategy for ensuring that
>> jars/dependencies are compatible between versions was "hope". If we want
>> to really verify this compatibility we would have to ensure that
>> "public" code does not transitively use any "non-public" dependencies.
>>
>> An alternative would be to say we don't support any cross-version
>> compatibility between Flink versions. If users want to use an older
>> connector they would have to copy the code, make sure it compiles
>> against a newer Flink version and then manage that themselves
>>
>> What do you think?
>>
>> Best,
>> Aljoscha
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-13586
>> [2] https://github.com/apache/flink/pull/12699#discussion_r442013514
>> [3] https://issues.apache.org/jira/browse/FLINK-17376
>> [4] https://issues.apache.org/jira/browse/FLINK-15115
>>
>
>
Re: [DISCUSS] Cross-version compatibility guarantees of Flink Modules/Jars
Posted by Konstantin Knauf <kn...@apache.org>.
Hi Aljoscha,
Thank you for bringing this up. IMHO the situation is different for minor &
patch version upgrades.
1) I don't think we need to provide any guarantees across Flink minor
versions (e.g. 1.10.1 -> 1.11.0). It seems reasonable to expect users to
recompile their user JARs when upgrading the cluster to the next minor
version of Apache Flink.
2) We should aim for compatibility across patch versions (1.10.0 ->
1.10.1). Being able to apply a patch to the Flink runtime/framework without
updating the dependencies of each user jar running on the cluster seems
really useful to me.
According to the discussion in [1] around stability guarantees for
@PublicEvolving, this would mean that connectors can use @Public
and @PublicEvolving classes, but not @Internal classes, right? This
generally seems reasonable to me. If we want to make it easy for our users,
we need to aim for stable interfaces anyway.
Cheers,
Konstantin
[]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stability-guarantees-for-PublicEvolving-classes-tp41459.html
On Tue, Jun 23, 2020 at 3:32 PM Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
>
> this has come up a few times now and I think we need to discuss the
> guarantees that we want to officially give for this. What I mean by
> cross-version compatibility is using, say, a Flink 1.10 Kafka connector
> dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
> 1.10.1. In the past, this hast mostly worked. I think this was largely
> by accident, though.
>
> The problem is that connectors, which might be annotated as @Public can
> use classes internally which are annotated as @Internal or
> @PublicEvolving. If those internal dependencies change, then the
> connector jar can not be used for different versions. This has happened
> at least in [1] and [2], where [2] was caused by the interplay of [3]
> and [4]. The initial release note on [4] was saying that the Kafka 0.9
> connector can be used with Flink 1.11, but this was rendered wrong by
> [3]. (Also, sorry for all the []s ...)
>
> What should we do about it? So far our strategy for ensuring that
> jars/dependencies are compatible between versions was "hope". If we want
> to really verify this compatibility we would have to ensure that
> "public" code does not transitively use any "non-public" dependencies.
>
> An alternative would be to say we don't support any cross-version
> compatibility between Flink versions. If users want to use an older
> connector they would have to copy the code, make sure it compiles
> against a newer Flink version and then manage that themselves
>
> What do you think?
>
> Best,
> Aljoscha
>
> [1] https://issues.apache.org/jira/browse/FLINK-13586
> [2] https://github.com/apache/flink/pull/12699#discussion_r442013514
> [3] https://issues.apache.org/jira/browse/FLINK-17376
> [4] https://issues.apache.org/jira/browse/FLINK-15115
>
--
Konstantin Knauf
https://twitter.com/snntrable
https://github.com/knaufk