You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Martijn Visser <ma...@apache.org> on 2022/10/04 11:58:17 UTC
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Hi Marton,
You're making a good point, I originally wanted to include already the User
mailing list to get their feedback but forgot to do so. I'll do some more
outreach via other channels as well.
@Users of Flink, I've made a proposal to deprecate and remove Scala API
support in a future version of Flink. Your feedback on this topic is very
much appreciated.
Regarding the large Scala codebase for Flink, a potential alternative could
be to have a wrapper for all Java APIs that makes them available as Scala
APIs. However, this still requires Scala maintainers and I don't think that
we currently have those in our community. The easiest solution for them
would be to use the Java APIs directly. Yes it would involve work, but we
won't actually be able to remove the Scala APIs until Flink 2.0 so there's
still time for that :)
Best regards,
Martijn
On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
wrote:
> Hi Martjin,
>
> Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
> considerable maintenance overhead and key improvements (like 2.13 or 2.12.8
> supports) are hanging stale. With that said before we make this move we
> should attempt to understand the userbase affected.
> A quick Slack and user mailing list search does return quite a bit of
> results for scala (admittedly a cursory look at them suggest that many of
> them have to do with missing features in Scala that exist in Java or Scala
> versions). I would love to see some polls on this topic, we could also use
> the Flink twitter handle to ask the community about this.
>
> I am aware of users having large existing Scala codebases for Flink. This
> move would pose a very large effort on them, as they would need to rewrite
> much of their existing code. What are the alternatives in your opinion,
> Martjin?
>
> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a discussion thread on FLIP-265 Deprecate and remove
> > Scala API support. Please take a look at
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
> > and provide your feedback.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com.INVALID>.
Hey Martijn,
Well, as a user I think that Scala API still adds a tremendous value, with
all its issues. But I'm not a committer and I don't know what effort it
takes to keep maintaining it... so I prepare for the worst :)
Regarding the proposed timeline, I don't know all the specifics around
breaking compatibility changes, so it's hard for me to say, but I'd love to
see Java 17 before Scala is fully removed from the project.
On Tue, Oct 4, 2022 at 3:07 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Yaroslav,
>
> If I could summarize your suggestion, would it mean that you would only be
> in favour of dropping Scala API support if we introduce Java 17 support
> exactly at the same time (say Flink 2.0). I was first thinking that an
> alternative would be to have a Flink 2.0 which supports Java 17 while
> keeping the Scala APIs (and postpone dropping those in Flink 3.0), but that
> wouldn't make much sense since we would have to break savepoint
> compatibility for Scala users then anyway.
>
> However, since we're already talking about when/how the APIs could be
> dropped, does that mean that you agree with the idea to deprecate and
> remove the Scala APIs?
>
> Best regards,
>
> Martijn
>
>
> On Tue, Oct 4, 2022 at 2:41 PM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> The 2.0 argument makes sense (I agree it's easier to introduce more
>> breaking changes in one major release), but I think my comment about Java
>> 17 also makes sense in this case: 1) easier to introduce because breaking
>> changes are possible 2) you'd need to give some syntax sugar as an
>> alternative after removing Scala.
>>
>> On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi Yaroslav,
>>>
>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>>> prerequisite, we would have to break compatibility already since Scala
>>> 2.12.7 doesn't compile on Java 17 [1].
>>>
>>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>>> version, would that still impact you a lot? I do imagine that if we get to
>>> a Flink 2.0 version there would be more breaking involved anyway. The
>>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>>> that new APIs would only be available in Java, but since these don't exist
>>> yet there would be no refactoring involved. I can imagine that we might
>>> change something in an existing API, but that would have certain
>>> compatibility guarantees already (depending if it's
>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>> think it would be smaller refactoring.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>
>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>>> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>
>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>>> expressions and records could simplify the migration quite a bit. Would you
>>>> consider adding it to the FLIP?
>>>>
>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>
>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>> very young and I don't think they will attract more scala developers as
>>>>> Flink could just because they are external projects. It will be a big issue
>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>> aware of the effort from now on and would better not count on those scala
>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>
>>>>> Best regards,
>>>>> Jing
>>>>>
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>> martijnvisser@apache.org> wrote:
>>>>>
>>>>>> Hi Marton,
>>>>>>
>>>>>> You're making a good point, I originally wanted to include already
>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>> some more outreach via other channels as well.
>>>>>>
>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>> very much appreciated.
>>>>>>
>>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>>> think that we currently have those in our community. The easiest solution
>>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>>> there's still time for that :)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martjin,
>>>>>>>
>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>>> poses
>>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>>> 2.12.8
>>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>>> we
>>>>>>> should attempt to understand the userbase affected.
>>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>> many of
>>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>>> Scala
>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>> also use
>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>
>>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>>> This
>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>> rewrite
>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>> opinion,
>>>>>>> Martjin?
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>> martijnvisser@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi everyone,
>>>>>>> >
>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>>> remove
>>>>>>> > Scala API support. Please take a look at
>>>>>>> >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>> > and provide your feedback.
>>>>>>> >
>>>>>>> > Best regards,
>>>>>>> >
>>>>>>> > Martijn
>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>> > https://github.com/MartijnVisser
>>>>>>> >
>>>>>>>
>>>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Hey Martijn,
Well, as a user I think that Scala API still adds a tremendous value, with
all its issues. But I'm not a committer and I don't know what effort it
takes to keep maintaining it... so I prepare for the worst :)
Regarding the proposed timeline, I don't know all the specifics around
breaking compatibility changes, so it's hard for me to say, but I'd love to
see Java 17 before Scala is fully removed from the project.
On Tue, Oct 4, 2022 at 3:07 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Yaroslav,
>
> If I could summarize your suggestion, would it mean that you would only be
> in favour of dropping Scala API support if we introduce Java 17 support
> exactly at the same time (say Flink 2.0). I was first thinking that an
> alternative would be to have a Flink 2.0 which supports Java 17 while
> keeping the Scala APIs (and postpone dropping those in Flink 3.0), but that
> wouldn't make much sense since we would have to break savepoint
> compatibility for Scala users then anyway.
>
> However, since we're already talking about when/how the APIs could be
> dropped, does that mean that you agree with the idea to deprecate and
> remove the Scala APIs?
>
> Best regards,
>
> Martijn
>
>
> On Tue, Oct 4, 2022 at 2:41 PM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> The 2.0 argument makes sense (I agree it's easier to introduce more
>> breaking changes in one major release), but I think my comment about Java
>> 17 also makes sense in this case: 1) easier to introduce because breaking
>> changes are possible 2) you'd need to give some syntax sugar as an
>> alternative after removing Scala.
>>
>> On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi Yaroslav,
>>>
>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>>> prerequisite, we would have to break compatibility already since Scala
>>> 2.12.7 doesn't compile on Java 17 [1].
>>>
>>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>>> version, would that still impact you a lot? I do imagine that if we get to
>>> a Flink 2.0 version there would be more breaking involved anyway. The
>>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>>> that new APIs would only be available in Java, but since these don't exist
>>> yet there would be no refactoring involved. I can imagine that we might
>>> change something in an existing API, but that would have certain
>>> compatibility guarantees already (depending if it's
>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>> think it would be smaller refactoring.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>
>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>>> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>
>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>>> expressions and records could simplify the migration quite a bit. Would you
>>>> consider adding it to the FLIP?
>>>>
>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>
>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>> very young and I don't think they will attract more scala developers as
>>>>> Flink could just because they are external projects. It will be a big issue
>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>> aware of the effort from now on and would better not count on those scala
>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>
>>>>> Best regards,
>>>>> Jing
>>>>>
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>> martijnvisser@apache.org> wrote:
>>>>>
>>>>>> Hi Marton,
>>>>>>
>>>>>> You're making a good point, I originally wanted to include already
>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>> some more outreach via other channels as well.
>>>>>>
>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>> very much appreciated.
>>>>>>
>>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>>> think that we currently have those in our community. The easiest solution
>>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>>> there's still time for that :)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martjin,
>>>>>>>
>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>>> poses
>>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>>> 2.12.8
>>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>>> we
>>>>>>> should attempt to understand the userbase affected.
>>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>> many of
>>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>>> Scala
>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>> also use
>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>
>>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>>> This
>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>> rewrite
>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>> opinion,
>>>>>>> Martjin?
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>> martijnvisser@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi everyone,
>>>>>>> >
>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>>> remove
>>>>>>> > Scala API support. Please take a look at
>>>>>>> >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>> > and provide your feedback.
>>>>>>> >
>>>>>>> > Best regards,
>>>>>>> >
>>>>>>> > Martijn
>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>> > https://github.com/MartijnVisser
>>>>>>> >
>>>>>>>
>>>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi Yaroslav,
If I could summarize your suggestion, would it mean that you would only be
in favour of dropping Scala API support if we introduce Java 17 support
exactly at the same time (say Flink 2.0). I was first thinking that an
alternative would be to have a Flink 2.0 which supports Java 17 while
keeping the Scala APIs (and postpone dropping those in Flink 3.0), but that
wouldn't make much sense since we would have to break savepoint
compatibility for Scala users then anyway.
However, since we're already talking about when/how the APIs could be
dropped, does that mean that you agree with the idea to deprecate and
remove the Scala APIs?
Best regards,
Martijn
On Tue, Oct 4, 2022 at 2:41 PM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Martijn,
>
> The 2.0 argument makes sense (I agree it's easier to introduce more
> breaking changes in one major release), but I think my comment about Java
> 17 also makes sense in this case: 1) easier to introduce because breaking
> changes are possible 2) you'd need to give some syntax sugar as an
> alternative after removing Scala.
>
> On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>> prerequisite, we would have to break compatibility already since Scala
>> 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>> version, would that still impact you a lot? I do imagine that if we get to
>> a Flink 2.0 version there would be more breaking involved anyway. The
>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>> that new APIs would only be available in Java, but since these don't exist
>> yet there would be no refactoring involved. I can imagine that we might
>> change something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen there, I
>> think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hi Martijn,
>>>
>>> As a Scala user, this change would affect me a lot and I'm not looking
>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>
>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>> expressions and records could simplify the migration quite a bit. Would you
>>> consider adding it to the FLIP?
>>>
>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>
>>>> Since both scala extension projects mentioned in the FLIP are still
>>>> very young and I don't think they will attract more scala developers as
>>>> Flink could just because they are external projects. It will be a big issue
>>>> for users who have to rewrite their large codebases. Those users should be
>>>> aware of the effort from now on and would better not count on those scala
>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>>
>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Marton,
>>>>>
>>>>> You're making a good point, I originally wanted to include already the
>>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>>> more outreach via other channels as well.
>>>>>
>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>> very much appreciated.
>>>>>
>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>> think that we currently have those in our community. The easiest solution
>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>> there's still time for that :)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>> balassi.marton@gmail.com> wrote:
>>>>>
>>>>>> Hi Martjin,
>>>>>>
>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>> poses
>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>> 2.12.8
>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>> we
>>>>>> should attempt to understand the userbase affected.
>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>> many of
>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>> Scala
>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>> also use
>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>
>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>> This
>>>>>> move would pose a very large effort on them, as they would need to
>>>>>> rewrite
>>>>>> much of their existing code. What are the alternatives in your
>>>>>> opinion,
>>>>>> Martjin?
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>> martijnvisser@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi everyone,
>>>>>> >
>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>> remove
>>>>>> > Scala API support. Please take a look at
>>>>>> >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>> > and provide your feedback.
>>>>>> >
>>>>>> > Best regards,
>>>>>> >
>>>>>> > Martijn
>>>>>> > https://twitter.com/MartijnVisser82
>>>>>> > https://github.com/MartijnVisser
>>>>>> >
>>>>>>
>>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi Yaroslav,
If I could summarize your suggestion, would it mean that you would only be
in favour of dropping Scala API support if we introduce Java 17 support
exactly at the same time (say Flink 2.0). I was first thinking that an
alternative would be to have a Flink 2.0 which supports Java 17 while
keeping the Scala APIs (and postpone dropping those in Flink 3.0), but that
wouldn't make much sense since we would have to break savepoint
compatibility for Scala users then anyway.
However, since we're already talking about when/how the APIs could be
dropped, does that mean that you agree with the idea to deprecate and
remove the Scala APIs?
Best regards,
Martijn
On Tue, Oct 4, 2022 at 2:41 PM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Martijn,
>
> The 2.0 argument makes sense (I agree it's easier to introduce more
> breaking changes in one major release), but I think my comment about Java
> 17 also makes sense in this case: 1) easier to introduce because breaking
> changes are possible 2) you'd need to give some syntax sugar as an
> alternative after removing Scala.
>
> On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>> prerequisite, we would have to break compatibility already since Scala
>> 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>> version, would that still impact you a lot? I do imagine that if we get to
>> a Flink 2.0 version there would be more breaking involved anyway. The
>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>> that new APIs would only be available in Java, but since these don't exist
>> yet there would be no refactoring involved. I can imagine that we might
>> change something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen there, I
>> think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hi Martijn,
>>>
>>> As a Scala user, this change would affect me a lot and I'm not looking
>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>
>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>> expressions and records could simplify the migration quite a bit. Would you
>>> consider adding it to the FLIP?
>>>
>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>
>>>> Since both scala extension projects mentioned in the FLIP are still
>>>> very young and I don't think they will attract more scala developers as
>>>> Flink could just because they are external projects. It will be a big issue
>>>> for users who have to rewrite their large codebases. Those users should be
>>>> aware of the effort from now on and would better not count on those scala
>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>>
>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Marton,
>>>>>
>>>>> You're making a good point, I originally wanted to include already the
>>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>>> more outreach via other channels as well.
>>>>>
>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>> very much appreciated.
>>>>>
>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>> think that we currently have those in our community. The easiest solution
>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>> there's still time for that :)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>> balassi.marton@gmail.com> wrote:
>>>>>
>>>>>> Hi Martjin,
>>>>>>
>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>> poses
>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>> 2.12.8
>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>> we
>>>>>> should attempt to understand the userbase affected.
>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>> many of
>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>> Scala
>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>> also use
>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>
>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>> This
>>>>>> move would pose a very large effort on them, as they would need to
>>>>>> rewrite
>>>>>> much of their existing code. What are the alternatives in your
>>>>>> opinion,
>>>>>> Martjin?
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>> martijnvisser@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi everyone,
>>>>>> >
>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>> remove
>>>>>> > Scala API support. Please take a look at
>>>>>> >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>> > and provide your feedback.
>>>>>> >
>>>>>> > Best regards,
>>>>>> >
>>>>>> > Martijn
>>>>>> > https://twitter.com/MartijnVisser82
>>>>>> > https://github.com/MartijnVisser
>>>>>> >
>>>>>>
>>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Hi Martijn,
The 2.0 argument makes sense (I agree it's easier to introduce more
breaking changes in one major release), but I think my comment about Java
17 also makes sense in this case: 1) easier to introduce because breaking
changes are possible 2) you'd need to give some syntax sugar as an
alternative after removing Scala.
On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Yaroslav,
>
> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
> prerequisite, we would have to break compatibility already since Scala
> 2.12.7 doesn't compile on Java 17 [1].
>
> Given that we can only remove Scala APIs with the next major Flink (2.0)
> version, would that still impact you a lot? I do imagine that if we get to
> a Flink 2.0 version there would be more breaking involved anyway. The
> biggest consequence of deprecating support for Scala in Flink 1.x would be
> that new APIs would only be available in Java, but since these don't exist
> yet there would be no refactoring involved. I can imagine that we might
> change something in an existing API, but that would have certain
> compatibility guarantees already (depending if it's
> Public/PublicEvolving/Experimental). If a change would happen there, I
> think it would be smaller refactoring.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-25000
>
> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm not looking
>> forward to rewriting my codebase, and it's not even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite (
>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>> expressions and records could simplify the migration quite a bit. Would you
>> consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>
>>> Hi Martijn,
>>>
>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>
>>> Since both scala extension projects mentioned in the FLIP are still very
>>> young and I don't think they will attract more scala developers as Flink
>>> could just because they are external projects. It will be a big issue for
>>> users who have to rewrite their large codebases. Those users should be
>>> aware of the effort from now on and would better not count on those scala
>>> extension projects and prepare their migration plan before Flink 2.0.
>>>
>>> Best regards,
>>> Jing
>>>
>>>
>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>>> Hi Marton,
>>>>
>>>> You're making a good point, I originally wanted to include already the
>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>> more outreach via other channels as well.
>>>>
>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>>> support in a future version of Flink. Your feedback on this topic is very
>>>> much appreciated.
>>>>
>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>> think that we currently have those in our community. The easiest solution
>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>> there's still time for that :)
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Martjin,
>>>>>
>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>> poses
>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>> 2.12.8
>>>>> supports) are hanging stale. With that said before we make this move we
>>>>> should attempt to understand the userbase affected.
>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>> results for scala (admittedly a cursory look at them suggest that many
>>>>> of
>>>>> them have to do with missing features in Scala that exist in Java or
>>>>> Scala
>>>>> versions). I would love to see some polls on this topic, we could also
>>>>> use
>>>>> the Flink twitter handle to ask the community about this.
>>>>>
>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>> This
>>>>> move would pose a very large effort on them, as they would need to
>>>>> rewrite
>>>>> much of their existing code. What are the alternatives in your opinion,
>>>>> Martjin?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>> martijnvisser@apache.org>
>>>>> wrote:
>>>>>
>>>>> > Hi everyone,
>>>>> >
>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>> remove
>>>>> > Scala API support. Please take a look at
>>>>> >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>> > and provide your feedback.
>>>>> >
>>>>> > Best regards,
>>>>> >
>>>>> > Martijn
>>>>> > https://twitter.com/MartijnVisser82
>>>>> > https://github.com/MartijnVisser
>>>>> >
>>>>>
>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
I also noticed that we two replies in a separate thread on the User mailing
list, which can be found at
https://lists.apache.org/thread/m5ntl3cj81wg7frbfqg9v75c7hqnxtls.
I've included Clayton and David in this email, to at least centralize the
conversation once more :)
On Wed, Oct 5, 2022 at 11:36 AM Martijn Visser <ma...@apache.org>
wrote:
> @Maciek
>
> I saw that I missed replying to your question:
>
> > Could you please remind what was the conclusion of discussion on
> upgrading Scala to 2.12.15/16?
> > https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
> couldn't find any follow-up vote?
>
> There is a vote thread, but that never got enough votes. See
> https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655
>
> > If it's acceptable to break binary compatibility by such an upgrade,
> then upgrading to JDK17 before 2.0 will be doable?
>
> I'm not sure, because I don't think a discussion and vote has been made
> yet if upgrading JDK17 can/will be done in a Flink 1.0 release or if it
> requires a 2.0 release. It was mentioned in the original discussion thread
> on dropping Java 8 support within 2/3 releases, but if I recall correctly
> there was no discussion yet on when Java 17 support would be added [1].
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx
>
> On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <ga...@datadome.co>
> wrote:
>
>> > I'm curious what target Scala versions people are currently interested
>> in.
>> > I would've expected that everyone wants to migrate to Scala 3, for
>> which several wrapper projects around Flink already exist
>>
>> The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not
>> sure I would move my team to Scala 3 right now (I'm currently toying with
>> it on a personal project). In addition, moving to Scala 3 is not completely
>> free - you have to do some rewrites, and developers will need some
>> adaptation time. Scala 2.13 is another thing entirely, we've wanted to
>> migrate for a long while.
>>
>> On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> > It's possible that for the sake of the Scala API, we would
>>> occasionally require some changes in the Java API. As long as those changes
>>> are not detrimental to Java users, they should be considered.
>>>
>>> That's exactly the model we're trying to get to. Don't fix
>>> scala-specific issues with scala code, but rather on the Java side as much
>>> as possible which could also benefit other JVM languages (e.g., Kotlin).
>>>
>>> > A question regarding the Flink wrapper: would it be possible to keep
>>> it under the Flink project's umbrella? Or does it need to be a completely
>>> separate structure? I'm not aware of the full organizational implications
>>> of this, I'm afraid.
>>>
>>> Technically it can be under the Flink umbrella, but then Flink would
>>> still be (at least for a while) be the bottleneck because we'd have to
>>> review any changes coming in.
>>> That would only improve once several new committers were added to take
>>> care of this project.
>>> (In the end you'd just split Flink and the Scala API _codebases_, but
>>> achieve little else)
>>>
>>> > And if that is what it takes to move beyond Scala 2.12.7… This has
>>> been a big pain point for us.
>>>
>>> I'm curious what target Scala versions people are currently interested
>>> in.
>>> I would've expected that everyone wants to migrate to Scala 3, for which
>>> several wrapper projects around Flink already exist.
>>>
>>> On 05/10/2022 12:35, Gaël Renoux wrote:
>>>
>>> Hello everyone,
>>>
>>> I've already answered a bit on Twitter, I'll develop my thoughts a bit
>>> here. For context, my company (DataDome) has a significant codebase on
>>> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
>>> an enthusiastic Scala developer (I don't think I'd like moving back to
>>> Java)
>>>
>>> Given that, I think separating the Scala support from Flink is actually
>>> a good move long term. We would then have a full-Java Flink, and a separate
>>> Scala wrapper. It would help a lot in solving the skills issue: Flink
>>> maintainers would no longer need to be fluent in Scala, and maintainers of
>>> the Scala wrapper would not need a deep knowledge of Flink's inner
>>> workings, just the API would be sufficient. And if that is what it takes to
>>> move beyond Scala 2.12.7… This has been a big pain point for us.
>>>
>>> I'm not too worried about finding contributors for the Scala wrapper.
>>> Within my company, we have developed additional wrappers and extension
>>> methods (for parts where we felt the Flink Scala API was insufficient), and
>>> we've been looking at ways we could contribute back. What held us back was
>>> our lack of knowledge of the full Flink environment (we're only using the
>>> Scala Datastream API). I don't think we're the only ones in that situation.
>>> One major point, though, is that Flink developers would not be completely
>>> rid of us ;-) It's possible that for the sake of the Scala API, we would
>>> occasionally require some changes in the Java API. As long as those changes
>>> are not detrimental to Java users, they should be considered.
>>>
>>> A question regarding the Flink wrapper: would it be possible to keep it
>>> under the Flink project's umbrella? Or does it need to be a completely
>>> separate structure? I'm not aware of the full organizational implications
>>> of this, I'm afraid.
>>>
>>> Finally, the hard part would be the migration to the new version. My
>>> dream solution would be to have the existing Scala API be entirely
>>> converted into a Scala wrapper over the Java API. That way, migration would
>>> be pretty minimal: add a dependency, change the imports for the Scala API,
>>> and we're done. However, even starting from the existing flink4s project,
>>> that's still quite a lot of work. So, more realistically, I'd settle for at
>>> least a partial implementation. We would have some broken code that we
>>> could fix, but at the very least I'd like the basic DataStream functions
>>> (process, uid, name…) to be available.
>>>
>>> Thanks for all the work that went into making Flink what it is!
>>>
>>>
>>> Gaël Renoux - Lead R&D Engineer
>>> E - gael.renoux@datadome.co
>>> W - www.datadome.co
>>>
>>>
>>>
>>> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> Could you please remind what was the conclusion of discussion on
>>>> upgrading Scala to 2.12.15/16?
>>>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>>>> couldn't find any follow-up vote?
>>>>
>>>> If it's acceptable to break binary compatibility by such an upgrade,
>>>> then upgrading to JDK17 before 2.0 will be doable?
>>>>
>>>>
>>>> thanks,
>>>>
>>>> maciek
>>>>
>>>>
>>>> On 04.10.2022 18:21, Martijn Visser wrote:
>>>>
>>>> Hi Yaroslav,
>>>>
>>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as
>>>> a prerequisite, we would have to break compatibility already since Scala
>>>> 2.12.7 doesn't compile on Java 17 [1].
>>>>
>>>> Given that we can only remove Scala APIs with the next major Flink
>>>> (2.0) version, would that still impact you a lot? I do imagine that if we
>>>> get to a Flink 2.0 version there would be more breaking involved anyway.
>>>> The biggest consequence of deprecating support for Scala in Flink 1.x would
>>>> be that new APIs would only be available in Java, but since these don't
>>>> exist yet there would be no refactoring involved. I can imagine that we
>>>> might change something in an existing API, but that would have certain
>>>> compatibility guarantees already (depending if it's
>>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>>> think it would be smaller refactoring.
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>>
>>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <
>>>> yaroslav@goldsky.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>>
>>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like
>>>>> switch expressions and records could simplify the migration quite a bit.
>>>>> Would you consider adding it to the FLIP?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>>
>>>>>> Hi Martijn,
>>>>>>
>>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>>
>>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>>> very young and I don't think they will attract more scala developers as
>>>>>> Flink could just because they are external projects. It will be a big issue
>>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>>> aware of the effort from now on and would better not count on those scala
>>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>>
>>>>>> Best regards,
>>>>>> Jing
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>>> martijnvisser@apache.org> wrote:
>>>>>>
>>>>>>> Hi Marton,
>>>>>>>
>>>>>>> You're making a good point, I originally wanted to include already
>>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>>> some more outreach via other channels as well.
>>>>>>>
>>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>>> very much appreciated.
>>>>>>>
>>>>>>> Regarding the large Scala codebase for Flink, a potential
>>>>>>> alternative could be to have a wrapper for all Java APIs that makes them
>>>>>>> available as Scala APIs. However, this still requires Scala maintainers and
>>>>>>> I don't think that we currently have those in our community. The easiest
>>>>>>> solution for them would be to use the Java APIs directly. Yes it would
>>>>>>> involve work, but we won't actually be able to remove the Scala APIs until
>>>>>>> Flink 2.0 so there's still time for that :)
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Martijn
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Martjin,
>>>>>>>>
>>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that
>>>>>>>> Scala poses
>>>>>>>> considerable maintenance overhead and key improvements (like 2.13
>>>>>>>> or 2.12.8
>>>>>>>> supports) are hanging stale. With that said before we make this
>>>>>>>> move we
>>>>>>>> should attempt to understand the userbase affected.
>>>>>>>> A quick Slack and user mailing list search does return quite a bit
>>>>>>>> of
>>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>>> many of
>>>>>>>> them have to do with missing features in Scala that exist in Java
>>>>>>>> or Scala
>>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>>> also use
>>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>>
>>>>>>>> I am aware of users having large existing Scala codebases for
>>>>>>>> Flink. This
>>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>>> rewrite
>>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>>> opinion,
>>>>>>>> Martjin?
>>>>>>>>
>>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>>> martijnvisser@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi everyone,
>>>>>>>> >
>>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate
>>>>>>>> and remove
>>>>>>>> > Scala API support. Please take a look at
>>>>>>>> >
>>>>>>>> >
>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>>> > and provide your feedback.
>>>>>>>> >
>>>>>>>> > Best regards,
>>>>>>>> >
>>>>>>>> > Martijn
>>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>>> > https://github.com/MartijnVisser
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
I also noticed that we two replies in a separate thread on the User mailing
list, which can be found at
https://lists.apache.org/thread/m5ntl3cj81wg7frbfqg9v75c7hqnxtls.
I've included Clayton and David in this email, to at least centralize the
conversation once more :)
On Wed, Oct 5, 2022 at 11:36 AM Martijn Visser <ma...@apache.org>
wrote:
> @Maciek
>
> I saw that I missed replying to your question:
>
> > Could you please remind what was the conclusion of discussion on
> upgrading Scala to 2.12.15/16?
> > https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
> couldn't find any follow-up vote?
>
> There is a vote thread, but that never got enough votes. See
> https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655
>
> > If it's acceptable to break binary compatibility by such an upgrade,
> then upgrading to JDK17 before 2.0 will be doable?
>
> I'm not sure, because I don't think a discussion and vote has been made
> yet if upgrading JDK17 can/will be done in a Flink 1.0 release or if it
> requires a 2.0 release. It was mentioned in the original discussion thread
> on dropping Java 8 support within 2/3 releases, but if I recall correctly
> there was no discussion yet on when Java 17 support would be added [1].
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx
>
> On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <ga...@datadome.co>
> wrote:
>
>> > I'm curious what target Scala versions people are currently interested
>> in.
>> > I would've expected that everyone wants to migrate to Scala 3, for
>> which several wrapper projects around Flink already exist
>>
>> The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not
>> sure I would move my team to Scala 3 right now (I'm currently toying with
>> it on a personal project). In addition, moving to Scala 3 is not completely
>> free - you have to do some rewrites, and developers will need some
>> adaptation time. Scala 2.13 is another thing entirely, we've wanted to
>> migrate for a long while.
>>
>> On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> > It's possible that for the sake of the Scala API, we would
>>> occasionally require some changes in the Java API. As long as those changes
>>> are not detrimental to Java users, they should be considered.
>>>
>>> That's exactly the model we're trying to get to. Don't fix
>>> scala-specific issues with scala code, but rather on the Java side as much
>>> as possible which could also benefit other JVM languages (e.g., Kotlin).
>>>
>>> > A question regarding the Flink wrapper: would it be possible to keep
>>> it under the Flink project's umbrella? Or does it need to be a completely
>>> separate structure? I'm not aware of the full organizational implications
>>> of this, I'm afraid.
>>>
>>> Technically it can be under the Flink umbrella, but then Flink would
>>> still be (at least for a while) be the bottleneck because we'd have to
>>> review any changes coming in.
>>> That would only improve once several new committers were added to take
>>> care of this project.
>>> (In the end you'd just split Flink and the Scala API _codebases_, but
>>> achieve little else)
>>>
>>> > And if that is what it takes to move beyond Scala 2.12.7… This has
>>> been a big pain point for us.
>>>
>>> I'm curious what target Scala versions people are currently interested
>>> in.
>>> I would've expected that everyone wants to migrate to Scala 3, for which
>>> several wrapper projects around Flink already exist.
>>>
>>> On 05/10/2022 12:35, Gaël Renoux wrote:
>>>
>>> Hello everyone,
>>>
>>> I've already answered a bit on Twitter, I'll develop my thoughts a bit
>>> here. For context, my company (DataDome) has a significant codebase on
>>> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
>>> an enthusiastic Scala developer (I don't think I'd like moving back to
>>> Java)
>>>
>>> Given that, I think separating the Scala support from Flink is actually
>>> a good move long term. We would then have a full-Java Flink, and a separate
>>> Scala wrapper. It would help a lot in solving the skills issue: Flink
>>> maintainers would no longer need to be fluent in Scala, and maintainers of
>>> the Scala wrapper would not need a deep knowledge of Flink's inner
>>> workings, just the API would be sufficient. And if that is what it takes to
>>> move beyond Scala 2.12.7… This has been a big pain point for us.
>>>
>>> I'm not too worried about finding contributors for the Scala wrapper.
>>> Within my company, we have developed additional wrappers and extension
>>> methods (for parts where we felt the Flink Scala API was insufficient), and
>>> we've been looking at ways we could contribute back. What held us back was
>>> our lack of knowledge of the full Flink environment (we're only using the
>>> Scala Datastream API). I don't think we're the only ones in that situation.
>>> One major point, though, is that Flink developers would not be completely
>>> rid of us ;-) It's possible that for the sake of the Scala API, we would
>>> occasionally require some changes in the Java API. As long as those changes
>>> are not detrimental to Java users, they should be considered.
>>>
>>> A question regarding the Flink wrapper: would it be possible to keep it
>>> under the Flink project's umbrella? Or does it need to be a completely
>>> separate structure? I'm not aware of the full organizational implications
>>> of this, I'm afraid.
>>>
>>> Finally, the hard part would be the migration to the new version. My
>>> dream solution would be to have the existing Scala API be entirely
>>> converted into a Scala wrapper over the Java API. That way, migration would
>>> be pretty minimal: add a dependency, change the imports for the Scala API,
>>> and we're done. However, even starting from the existing flink4s project,
>>> that's still quite a lot of work. So, more realistically, I'd settle for at
>>> least a partial implementation. We would have some broken code that we
>>> could fix, but at the very least I'd like the basic DataStream functions
>>> (process, uid, name…) to be available.
>>>
>>> Thanks for all the work that went into making Flink what it is!
>>>
>>>
>>> Gaël Renoux - Lead R&D Engineer
>>> E - gael.renoux@datadome.co
>>> W - www.datadome.co
>>>
>>>
>>>
>>> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> Could you please remind what was the conclusion of discussion on
>>>> upgrading Scala to 2.12.15/16?
>>>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>>>> couldn't find any follow-up vote?
>>>>
>>>> If it's acceptable to break binary compatibility by such an upgrade,
>>>> then upgrading to JDK17 before 2.0 will be doable?
>>>>
>>>>
>>>> thanks,
>>>>
>>>> maciek
>>>>
>>>>
>>>> On 04.10.2022 18:21, Martijn Visser wrote:
>>>>
>>>> Hi Yaroslav,
>>>>
>>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as
>>>> a prerequisite, we would have to break compatibility already since Scala
>>>> 2.12.7 doesn't compile on Java 17 [1].
>>>>
>>>> Given that we can only remove Scala APIs with the next major Flink
>>>> (2.0) version, would that still impact you a lot? I do imagine that if we
>>>> get to a Flink 2.0 version there would be more breaking involved anyway.
>>>> The biggest consequence of deprecating support for Scala in Flink 1.x would
>>>> be that new APIs would only be available in Java, but since these don't
>>>> exist yet there would be no refactoring involved. I can imagine that we
>>>> might change something in an existing API, but that would have certain
>>>> compatibility guarantees already (depending if it's
>>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>>> think it would be smaller refactoring.
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>>
>>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <
>>>> yaroslav@goldsky.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>>
>>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like
>>>>> switch expressions and records could simplify the migration quite a bit.
>>>>> Would you consider adding it to the FLIP?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>>
>>>>>> Hi Martijn,
>>>>>>
>>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>>
>>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>>> very young and I don't think they will attract more scala developers as
>>>>>> Flink could just because they are external projects. It will be a big issue
>>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>>> aware of the effort from now on and would better not count on those scala
>>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>>
>>>>>> Best regards,
>>>>>> Jing
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>>> martijnvisser@apache.org> wrote:
>>>>>>
>>>>>>> Hi Marton,
>>>>>>>
>>>>>>> You're making a good point, I originally wanted to include already
>>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>>> some more outreach via other channels as well.
>>>>>>>
>>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>>> very much appreciated.
>>>>>>>
>>>>>>> Regarding the large Scala codebase for Flink, a potential
>>>>>>> alternative could be to have a wrapper for all Java APIs that makes them
>>>>>>> available as Scala APIs. However, this still requires Scala maintainers and
>>>>>>> I don't think that we currently have those in our community. The easiest
>>>>>>> solution for them would be to use the Java APIs directly. Yes it would
>>>>>>> involve work, but we won't actually be able to remove the Scala APIs until
>>>>>>> Flink 2.0 so there's still time for that :)
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Martijn
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Martjin,
>>>>>>>>
>>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that
>>>>>>>> Scala poses
>>>>>>>> considerable maintenance overhead and key improvements (like 2.13
>>>>>>>> or 2.12.8
>>>>>>>> supports) are hanging stale. With that said before we make this
>>>>>>>> move we
>>>>>>>> should attempt to understand the userbase affected.
>>>>>>>> A quick Slack and user mailing list search does return quite a bit
>>>>>>>> of
>>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>>> many of
>>>>>>>> them have to do with missing features in Scala that exist in Java
>>>>>>>> or Scala
>>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>>> also use
>>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>>
>>>>>>>> I am aware of users having large existing Scala codebases for
>>>>>>>> Flink. This
>>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>>> rewrite
>>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>>> opinion,
>>>>>>>> Martjin?
>>>>>>>>
>>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>>> martijnvisser@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi everyone,
>>>>>>>> >
>>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate
>>>>>>>> and remove
>>>>>>>> > Scala API support. Please take a look at
>>>>>>>> >
>>>>>>>> >
>>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>>> > and provide your feedback.
>>>>>>>> >
>>>>>>>> > Best regards,
>>>>>>>> >
>>>>>>>> > Martijn
>>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>>> > https://github.com/MartijnVisser
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
@Maciek
I saw that I missed replying to your question:
> Could you please remind what was the conclusion of discussion on
upgrading Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
couldn't find any follow-up vote?
There is a vote thread, but that never got enough votes. See
https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655
> If it's acceptable to break binary compatibility by such an upgrade, then
upgrading to JDK17 before 2.0 will be doable?
I'm not sure, because I don't think a discussion and vote has been made yet
if upgrading JDK17 can/will be done in a Flink 1.0 release or if it
requires a 2.0 release. It was mentioned in the original discussion thread
on dropping Java 8 support within 2/3 releases, but if I recall correctly
there was no discussion yet on when Java 17 support would be added [1].
Best regards,
Martijn
[1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx
On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <ga...@datadome.co> wrote:
> > I'm curious what target Scala versions people are currently interested
> in.
> > I would've expected that everyone wants to migrate to Scala 3, for which
> several wrapper projects around Flink already exist
>
> The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not
> sure I would move my team to Scala 3 right now (I'm currently toying with
> it on a personal project). In addition, moving to Scala 3 is not completely
> free - you have to do some rewrites, and developers will need some
> adaptation time. Scala 2.13 is another thing entirely, we've wanted to
> migrate for a long while.
>
> On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> > It's possible that for the sake of the Scala API, we would occasionally
>> require some changes in the Java API. As long as those changes are not
>> detrimental to Java users, they should be considered.
>>
>> That's exactly the model we're trying to get to. Don't fix scala-specific
>> issues with scala code, but rather on the Java side as much as possible
>> which could also benefit other JVM languages (e.g., Kotlin).
>>
>> > A question regarding the Flink wrapper: would it be possible to keep it
>> under the Flink project's umbrella? Or does it need to be a completely
>> separate structure? I'm not aware of the full organizational implications
>> of this, I'm afraid.
>>
>> Technically it can be under the Flink umbrella, but then Flink would
>> still be (at least for a while) be the bottleneck because we'd have to
>> review any changes coming in.
>> That would only improve once several new committers were added to take
>> care of this project.
>> (In the end you'd just split Flink and the Scala API _codebases_, but
>> achieve little else)
>>
>> > And if that is what it takes to move beyond Scala 2.12.7… This has been
>> a big pain point for us.
>>
>> I'm curious what target Scala versions people are currently interested in.
>> I would've expected that everyone wants to migrate to Scala 3, for which
>> several wrapper projects around Flink already exist.
>>
>> On 05/10/2022 12:35, Gaël Renoux wrote:
>>
>> Hello everyone,
>>
>> I've already answered a bit on Twitter, I'll develop my thoughts a bit
>> here. For context, my company (DataDome) has a significant codebase on
>> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
>> an enthusiastic Scala developer (I don't think I'd like moving back to
>> Java)
>>
>> Given that, I think separating the Scala support from Flink is actually a
>> good move long term. We would then have a full-Java Flink, and a separate
>> Scala wrapper. It would help a lot in solving the skills issue: Flink
>> maintainers would no longer need to be fluent in Scala, and maintainers of
>> the Scala wrapper would not need a deep knowledge of Flink's inner
>> workings, just the API would be sufficient. And if that is what it takes to
>> move beyond Scala 2.12.7… This has been a big pain point for us.
>>
>> I'm not too worried about finding contributors for the Scala wrapper.
>> Within my company, we have developed additional wrappers and extension
>> methods (for parts where we felt the Flink Scala API was insufficient), and
>> we've been looking at ways we could contribute back. What held us back was
>> our lack of knowledge of the full Flink environment (we're only using the
>> Scala Datastream API). I don't think we're the only ones in that situation.
>> One major point, though, is that Flink developers would not be completely
>> rid of us ;-) It's possible that for the sake of the Scala API, we would
>> occasionally require some changes in the Java API. As long as those changes
>> are not detrimental to Java users, they should be considered.
>>
>> A question regarding the Flink wrapper: would it be possible to keep it
>> under the Flink project's umbrella? Or does it need to be a completely
>> separate structure? I'm not aware of the full organizational implications
>> of this, I'm afraid.
>>
>> Finally, the hard part would be the migration to the new version. My
>> dream solution would be to have the existing Scala API be entirely
>> converted into a Scala wrapper over the Java API. That way, migration would
>> be pretty minimal: add a dependency, change the imports for the Scala API,
>> and we're done. However, even starting from the existing flink4s project,
>> that's still quite a lot of work. So, more realistically, I'd settle for at
>> least a partial implementation. We would have some broken code that we
>> could fix, but at the very least I'd like the basic DataStream functions
>> (process, uid, name…) to be available.
>>
>> Thanks for all the work that went into making Flink what it is!
>>
>>
>> Gaël Renoux - Lead R&D Engineer
>> E - gael.renoux@datadome.co
>> W - www.datadome.co
>>
>>
>>
>> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>>
>>> Hi Martin,
>>>
>>> Could you please remind what was the conclusion of discussion on
>>> upgrading Scala to 2.12.15/16?
>>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>>> couldn't find any follow-up vote?
>>>
>>> If it's acceptable to break binary compatibility by such an upgrade,
>>> then upgrading to JDK17 before 2.0 will be doable?
>>>
>>>
>>> thanks,
>>>
>>> maciek
>>>
>>>
>>> On 04.10.2022 18:21, Martijn Visser wrote:
>>>
>>> Hi Yaroslav,
>>>
>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>>> prerequisite, we would have to break compatibility already since Scala
>>> 2.12.7 doesn't compile on Java 17 [1].
>>>
>>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>>> version, would that still impact you a lot? I do imagine that if we get to
>>> a Flink 2.0 version there would be more breaking involved anyway. The
>>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>>> that new APIs would only be available in Java, but since these don't exist
>>> yet there would be no refactoring involved. I can imagine that we might
>>> change something in an existing API, but that would have certain
>>> compatibility guarantees already (depending if it's
>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>> think it would be smaller refactoring.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>
>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>>> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>
>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>>> expressions and records could simplify the migration quite a bit. Would you
>>>> consider adding it to the FLIP?
>>>>
>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>
>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>> very young and I don't think they will attract more scala developers as
>>>>> Flink could just because they are external projects. It will be a big issue
>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>> aware of the effort from now on and would better not count on those scala
>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>
>>>>> Best regards,
>>>>> Jing
>>>>>
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>> martijnvisser@apache.org> wrote:
>>>>>
>>>>>> Hi Marton,
>>>>>>
>>>>>> You're making a good point, I originally wanted to include already
>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>> some more outreach via other channels as well.
>>>>>>
>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>> very much appreciated.
>>>>>>
>>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>>> think that we currently have those in our community. The easiest solution
>>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>>> there's still time for that :)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martjin,
>>>>>>>
>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>>> poses
>>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>>> 2.12.8
>>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>>> we
>>>>>>> should attempt to understand the userbase affected.
>>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>> many of
>>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>>> Scala
>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>> also use
>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>
>>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>>> This
>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>> rewrite
>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>> opinion,
>>>>>>> Martjin?
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>> martijnvisser@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi everyone,
>>>>>>> >
>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>>> remove
>>>>>>> > Scala API support. Please take a look at
>>>>>>> >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>> > and provide your feedback.
>>>>>>> >
>>>>>>> > Best regards,
>>>>>>> >
>>>>>>> > Martijn
>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>> > https://github.com/MartijnVisser
>>>>>>> >
>>>>>>>
>>>>>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
@Maciek
I saw that I missed replying to your question:
> Could you please remind what was the conclusion of discussion on
upgrading Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
couldn't find any follow-up vote?
There is a vote thread, but that never got enough votes. See
https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655
> If it's acceptable to break binary compatibility by such an upgrade, then
upgrading to JDK17 before 2.0 will be doable?
I'm not sure, because I don't think a discussion and vote has been made yet
if upgrading JDK17 can/will be done in a Flink 1.0 release or if it
requires a 2.0 release. It was mentioned in the original discussion thread
on dropping Java 8 support within 2/3 releases, but if I recall correctly
there was no discussion yet on when Java 17 support would be added [1].
Best regards,
Martijn
[1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx
On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <ga...@datadome.co> wrote:
> > I'm curious what target Scala versions people are currently interested
> in.
> > I would've expected that everyone wants to migrate to Scala 3, for which
> several wrapper projects around Flink already exist
>
> The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not
> sure I would move my team to Scala 3 right now (I'm currently toying with
> it on a personal project). In addition, moving to Scala 3 is not completely
> free - you have to do some rewrites, and developers will need some
> adaptation time. Scala 2.13 is another thing entirely, we've wanted to
> migrate for a long while.
>
> On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> > It's possible that for the sake of the Scala API, we would occasionally
>> require some changes in the Java API. As long as those changes are not
>> detrimental to Java users, they should be considered.
>>
>> That's exactly the model we're trying to get to. Don't fix scala-specific
>> issues with scala code, but rather on the Java side as much as possible
>> which could also benefit other JVM languages (e.g., Kotlin).
>>
>> > A question regarding the Flink wrapper: would it be possible to keep it
>> under the Flink project's umbrella? Or does it need to be a completely
>> separate structure? I'm not aware of the full organizational implications
>> of this, I'm afraid.
>>
>> Technically it can be under the Flink umbrella, but then Flink would
>> still be (at least for a while) be the bottleneck because we'd have to
>> review any changes coming in.
>> That would only improve once several new committers were added to take
>> care of this project.
>> (In the end you'd just split Flink and the Scala API _codebases_, but
>> achieve little else)
>>
>> > And if that is what it takes to move beyond Scala 2.12.7… This has been
>> a big pain point for us.
>>
>> I'm curious what target Scala versions people are currently interested in.
>> I would've expected that everyone wants to migrate to Scala 3, for which
>> several wrapper projects around Flink already exist.
>>
>> On 05/10/2022 12:35, Gaël Renoux wrote:
>>
>> Hello everyone,
>>
>> I've already answered a bit on Twitter, I'll develop my thoughts a bit
>> here. For context, my company (DataDome) has a significant codebase on
>> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
>> an enthusiastic Scala developer (I don't think I'd like moving back to
>> Java)
>>
>> Given that, I think separating the Scala support from Flink is actually a
>> good move long term. We would then have a full-Java Flink, and a separate
>> Scala wrapper. It would help a lot in solving the skills issue: Flink
>> maintainers would no longer need to be fluent in Scala, and maintainers of
>> the Scala wrapper would not need a deep knowledge of Flink's inner
>> workings, just the API would be sufficient. And if that is what it takes to
>> move beyond Scala 2.12.7… This has been a big pain point for us.
>>
>> I'm not too worried about finding contributors for the Scala wrapper.
>> Within my company, we have developed additional wrappers and extension
>> methods (for parts where we felt the Flink Scala API was insufficient), and
>> we've been looking at ways we could contribute back. What held us back was
>> our lack of knowledge of the full Flink environment (we're only using the
>> Scala Datastream API). I don't think we're the only ones in that situation.
>> One major point, though, is that Flink developers would not be completely
>> rid of us ;-) It's possible that for the sake of the Scala API, we would
>> occasionally require some changes in the Java API. As long as those changes
>> are not detrimental to Java users, they should be considered.
>>
>> A question regarding the Flink wrapper: would it be possible to keep it
>> under the Flink project's umbrella? Or does it need to be a completely
>> separate structure? I'm not aware of the full organizational implications
>> of this, I'm afraid.
>>
>> Finally, the hard part would be the migration to the new version. My
>> dream solution would be to have the existing Scala API be entirely
>> converted into a Scala wrapper over the Java API. That way, migration would
>> be pretty minimal: add a dependency, change the imports for the Scala API,
>> and we're done. However, even starting from the existing flink4s project,
>> that's still quite a lot of work. So, more realistically, I'd settle for at
>> least a partial implementation. We would have some broken code that we
>> could fix, but at the very least I'd like the basic DataStream functions
>> (process, uid, name…) to be available.
>>
>> Thanks for all the work that went into making Flink what it is!
>>
>>
>> Gaël Renoux - Lead R&D Engineer
>> E - gael.renoux@datadome.co
>> W - www.datadome.co
>>
>>
>>
>> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>>
>>> Hi Martin,
>>>
>>> Could you please remind what was the conclusion of discussion on
>>> upgrading Scala to 2.12.15/16?
>>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>>> couldn't find any follow-up vote?
>>>
>>> If it's acceptable to break binary compatibility by such an upgrade,
>>> then upgrading to JDK17 before 2.0 will be doable?
>>>
>>>
>>> thanks,
>>>
>>> maciek
>>>
>>>
>>> On 04.10.2022 18:21, Martijn Visser wrote:
>>>
>>> Hi Yaroslav,
>>>
>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>>> prerequisite, we would have to break compatibility already since Scala
>>> 2.12.7 doesn't compile on Java 17 [1].
>>>
>>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>>> version, would that still impact you a lot? I do imagine that if we get to
>>> a Flink 2.0 version there would be more breaking involved anyway. The
>>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>>> that new APIs would only be available in Java, but since these don't exist
>>> yet there would be no refactoring involved. I can imagine that we might
>>> change something in an existing API, but that would have certain
>>> compatibility guarantees already (depending if it's
>>> Public/PublicEvolving/Experimental). If a change would happen there, I
>>> think it would be smaller refactoring.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>>
>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>>> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> As a Scala user, this change would affect me a lot and I'm not looking
>>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>>
>>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>>> expressions and records could simplify the migration quite a bit. Would you
>>>> consider adding it to the FLIP?
>>>>
>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>>
>>>>> Since both scala extension projects mentioned in the FLIP are still
>>>>> very young and I don't think they will attract more scala developers as
>>>>> Flink could just because they are external projects. It will be a big issue
>>>>> for users who have to rewrite their large codebases. Those users should be
>>>>> aware of the effort from now on and would better not count on those scala
>>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>>
>>>>> Best regards,
>>>>> Jing
>>>>>
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <
>>>>> martijnvisser@apache.org> wrote:
>>>>>
>>>>>> Hi Marton,
>>>>>>
>>>>>> You're making a good point, I originally wanted to include already
>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do
>>>>>> some more outreach via other channels as well.
>>>>>>
>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>>> very much appreciated.
>>>>>>
>>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>>> think that we currently have those in our community. The easiest solution
>>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>>> there's still time for that :)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>>> balassi.marton@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martjin,
>>>>>>>
>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>>> poses
>>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>>> 2.12.8
>>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>>> we
>>>>>>> should attempt to understand the userbase affected.
>>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>>> many of
>>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>>> Scala
>>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>>> also use
>>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>>
>>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>>> This
>>>>>>> move would pose a very large effort on them, as they would need to
>>>>>>> rewrite
>>>>>>> much of their existing code. What are the alternatives in your
>>>>>>> opinion,
>>>>>>> Martjin?
>>>>>>>
>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>>> martijnvisser@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi everyone,
>>>>>>> >
>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>>> remove
>>>>>>> > Scala API support. Please take a look at
>>>>>>> >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>>> > and provide your feedback.
>>>>>>> >
>>>>>>> > Best regards,
>>>>>>> >
>>>>>>> > Martijn
>>>>>>> > https://twitter.com/MartijnVisser82
>>>>>>> > https://github.com/MartijnVisser
>>>>>>> >
>>>>>>>
>>>>>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Gaël Renoux <ga...@datadome.co>.
> I'm curious what target Scala versions people are currently interested
in.
> I would've expected that everyone wants to migrate to Scala 3, for which
several wrapper projects around Flink already exist
The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not sure
I would move my team to Scala 3 right now (I'm currently toying with it on
a personal project). In addition, moving to Scala 3 is not completely free
- you have to do some rewrites, and developers will need some adaptation
time. Scala 2.13 is another thing entirely, we've wanted to migrate for a
long while.
On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org> wrote:
> > It's possible that for the sake of the Scala API, we would occasionally
> require some changes in the Java API. As long as those changes are not
> detrimental to Java users, they should be considered.
>
> That's exactly the model we're trying to get to. Don't fix scala-specific
> issues with scala code, but rather on the Java side as much as possible
> which could also benefit other JVM languages (e.g., Kotlin).
>
> > A question regarding the Flink wrapper: would it be possible to keep it
> under the Flink project's umbrella? Or does it need to be a completely
> separate structure? I'm not aware of the full organizational implications
> of this, I'm afraid.
>
> Technically it can be under the Flink umbrella, but then Flink would still
> be (at least for a while) be the bottleneck because we'd have to review any
> changes coming in.
> That would only improve once several new committers were added to take
> care of this project.
> (In the end you'd just split Flink and the Scala API _codebases_, but
> achieve little else)
>
> > And if that is what it takes to move beyond Scala 2.12.7… This has been
> a big pain point for us.
>
> I'm curious what target Scala versions people are currently interested in.
> I would've expected that everyone wants to migrate to Scala 3, for which
> several wrapper projects around Flink already exist.
>
> On 05/10/2022 12:35, Gaël Renoux wrote:
>
> Hello everyone,
>
> I've already answered a bit on Twitter, I'll develop my thoughts a bit
> here. For context, my company (DataDome) has a significant codebase on
> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
> an enthusiastic Scala developer (I don't think I'd like moving back to
> Java)
>
> Given that, I think separating the Scala support from Flink is actually a
> good move long term. We would then have a full-Java Flink, and a separate
> Scala wrapper. It would help a lot in solving the skills issue: Flink
> maintainers would no longer need to be fluent in Scala, and maintainers of
> the Scala wrapper would not need a deep knowledge of Flink's inner
> workings, just the API would be sufficient. And if that is what it takes to
> move beyond Scala 2.12.7… This has been a big pain point for us.
>
> I'm not too worried about finding contributors for the Scala wrapper.
> Within my company, we have developed additional wrappers and extension
> methods (for parts where we felt the Flink Scala API was insufficient), and
> we've been looking at ways we could contribute back. What held us back was
> our lack of knowledge of the full Flink environment (we're only using the
> Scala Datastream API). I don't think we're the only ones in that situation.
> One major point, though, is that Flink developers would not be completely
> rid of us ;-) It's possible that for the sake of the Scala API, we would
> occasionally require some changes in the Java API. As long as those changes
> are not detrimental to Java users, they should be considered.
>
> A question regarding the Flink wrapper: would it be possible to keep it
> under the Flink project's umbrella? Or does it need to be a completely
> separate structure? I'm not aware of the full organizational implications
> of this, I'm afraid.
>
> Finally, the hard part would be the migration to the new version. My dream
> solution would be to have the existing Scala API be entirely converted into
> a Scala wrapper over the Java API. That way, migration would be pretty
> minimal: add a dependency, change the imports for the Scala API, and we're
> done. However, even starting from the existing flink4s project, that's
> still quite a lot of work. So, more realistically, I'd settle for at least
> a partial implementation. We would have some broken code that we could fix,
> but at the very least I'd like the basic DataStream functions (process,
> uid, name…) to be available.
>
> Thanks for all the work that went into making Flink what it is!
>
>
> Gaël Renoux - Lead R&D Engineer
> E - gael.renoux@datadome.co
> W - www.datadome.co
>
>
>
> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>
>> Hi Martin,
>>
>> Could you please remind what was the conclusion of discussion on
>> upgrading Scala to 2.12.15/16?
>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>> couldn't find any follow-up vote?
>>
>> If it's acceptable to break binary compatibility by such an upgrade, then
>> upgrading to JDK17 before 2.0 will be doable?
>>
>>
>> thanks,
>>
>> maciek
>>
>>
>> On 04.10.2022 18:21, Martijn Visser wrote:
>>
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>> prerequisite, we would have to break compatibility already since Scala
>> 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>> version, would that still impact you a lot? I do imagine that if we get to
>> a Flink 2.0 version there would be more breaking involved anyway. The
>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>> that new APIs would only be available in Java, but since these don't exist
>> yet there would be no refactoring involved. I can imagine that we might
>> change something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen there, I
>> think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hi Martijn,
>>>
>>> As a Scala user, this change would affect me a lot and I'm not looking
>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>
>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>> expressions and records could simplify the migration quite a bit. Would you
>>> consider adding it to the FLIP?
>>>
>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>
>>>> Since both scala extension projects mentioned in the FLIP are still
>>>> very young and I don't think they will attract more scala developers as
>>>> Flink could just because they are external projects. It will be a big issue
>>>> for users who have to rewrite their large codebases. Those users should be
>>>> aware of the effort from now on and would better not count on those scala
>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>>
>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Marton,
>>>>>
>>>>> You're making a good point, I originally wanted to include already the
>>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>>> more outreach via other channels as well.
>>>>>
>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>> very much appreciated.
>>>>>
>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>> think that we currently have those in our community. The easiest solution
>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>> there's still time for that :)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>> balassi.marton@gmail.com> wrote:
>>>>>
>>>>>> Hi Martjin,
>>>>>>
>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>> poses
>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>> 2.12.8
>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>> we
>>>>>> should attempt to understand the userbase affected.
>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>> many of
>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>> Scala
>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>> also use
>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>
>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>> This
>>>>>> move would pose a very large effort on them, as they would need to
>>>>>> rewrite
>>>>>> much of their existing code. What are the alternatives in your
>>>>>> opinion,
>>>>>> Martjin?
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>> martijnvisser@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi everyone,
>>>>>> >
>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>> remove
>>>>>> > Scala API support. Please take a look at
>>>>>> >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>> > and provide your feedback.
>>>>>> >
>>>>>> > Best regards,
>>>>>> >
>>>>>> > Martijn
>>>>>> > https://twitter.com/MartijnVisser82
>>>>>> > https://github.com/MartijnVisser
>>>>>> >
>>>>>>
>>>>>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Gaël Renoux <ga...@datadome.co.INVALID>.
> I'm curious what target Scala versions people are currently interested
in.
> I would've expected that everyone wants to migrate to Scala 3, for which
several wrapper projects around Flink already exist
The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not sure
I would move my team to Scala 3 right now (I'm currently toying with it on
a personal project). In addition, moving to Scala 3 is not completely free
- you have to do some rewrites, and developers will need some adaptation
time. Scala 2.13 is another thing entirely, we've wanted to migrate for a
long while.
On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ch...@apache.org> wrote:
> > It's possible that for the sake of the Scala API, we would occasionally
> require some changes in the Java API. As long as those changes are not
> detrimental to Java users, they should be considered.
>
> That's exactly the model we're trying to get to. Don't fix scala-specific
> issues with scala code, but rather on the Java side as much as possible
> which could also benefit other JVM languages (e.g., Kotlin).
>
> > A question regarding the Flink wrapper: would it be possible to keep it
> under the Flink project's umbrella? Or does it need to be a completely
> separate structure? I'm not aware of the full organizational implications
> of this, I'm afraid.
>
> Technically it can be under the Flink umbrella, but then Flink would still
> be (at least for a while) be the bottleneck because we'd have to review any
> changes coming in.
> That would only improve once several new committers were added to take
> care of this project.
> (In the end you'd just split Flink and the Scala API _codebases_, but
> achieve little else)
>
> > And if that is what it takes to move beyond Scala 2.12.7… This has been
> a big pain point for us.
>
> I'm curious what target Scala versions people are currently interested in.
> I would've expected that everyone wants to migrate to Scala 3, for which
> several wrapper projects around Flink already exist.
>
> On 05/10/2022 12:35, Gaël Renoux wrote:
>
> Hello everyone,
>
> I've already answered a bit on Twitter, I'll develop my thoughts a bit
> here. For context, my company (DataDome) has a significant codebase on
> Scala Flink (around 110K LOC), having been using it since 2017. I myself am
> an enthusiastic Scala developer (I don't think I'd like moving back to
> Java)
>
> Given that, I think separating the Scala support from Flink is actually a
> good move long term. We would then have a full-Java Flink, and a separate
> Scala wrapper. It would help a lot in solving the skills issue: Flink
> maintainers would no longer need to be fluent in Scala, and maintainers of
> the Scala wrapper would not need a deep knowledge of Flink's inner
> workings, just the API would be sufficient. And if that is what it takes to
> move beyond Scala 2.12.7… This has been a big pain point for us.
>
> I'm not too worried about finding contributors for the Scala wrapper.
> Within my company, we have developed additional wrappers and extension
> methods (for parts where we felt the Flink Scala API was insufficient), and
> we've been looking at ways we could contribute back. What held us back was
> our lack of knowledge of the full Flink environment (we're only using the
> Scala Datastream API). I don't think we're the only ones in that situation.
> One major point, though, is that Flink developers would not be completely
> rid of us ;-) It's possible that for the sake of the Scala API, we would
> occasionally require some changes in the Java API. As long as those changes
> are not detrimental to Java users, they should be considered.
>
> A question regarding the Flink wrapper: would it be possible to keep it
> under the Flink project's umbrella? Or does it need to be a completely
> separate structure? I'm not aware of the full organizational implications
> of this, I'm afraid.
>
> Finally, the hard part would be the migration to the new version. My dream
> solution would be to have the existing Scala API be entirely converted into
> a Scala wrapper over the Java API. That way, migration would be pretty
> minimal: add a dependency, change the imports for the Scala API, and we're
> done. However, even starting from the existing flink4s project, that's
> still quite a lot of work. So, more realistically, I'd settle for at least
> a partial implementation. We would have some broken code that we could fix,
> but at the very least I'd like the basic DataStream functions (process,
> uid, name…) to be available.
>
> Thanks for all the work that went into making Flink what it is!
>
>
> Gaël Renoux - Lead R&D Engineer
> E - gael.renoux@datadome.co
> W - www.datadome.co
>
>
>
> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>
>> Hi Martin,
>>
>> Could you please remind what was the conclusion of discussion on
>> upgrading Scala to 2.12.15/16?
>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
>> couldn't find any follow-up vote?
>>
>> If it's acceptable to break binary compatibility by such an upgrade, then
>> upgrading to JDK17 before 2.0 will be doable?
>>
>>
>> thanks,
>>
>> maciek
>>
>>
>> On 04.10.2022 18:21, Martijn Visser wrote:
>>
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
>> prerequisite, we would have to break compatibility already since Scala
>> 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major Flink (2.0)
>> version, would that still impact you a lot? I do imagine that if we get to
>> a Flink 2.0 version there would be more breaking involved anyway. The
>> biggest consequence of deprecating support for Scala in Flink 1.x would be
>> that new APIs would only be available in Java, but since these don't exist
>> yet there would be no refactoring involved. I can imagine that we might
>> change something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen there, I
>> think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hi Martijn,
>>>
>>> As a Scala user, this change would affect me a lot and I'm not looking
>>> forward to rewriting my codebase, and it's not even a very large one :)
>>>
>>> I'd like to suggest supporting Java 17 as a prerequisite (
>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>>> expressions and records could simplify the migration quite a bit. Would you
>>> consider adding it to the FLIP?
>>>
>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>>
>>>> Since both scala extension projects mentioned in the FLIP are still
>>>> very young and I don't think they will attract more scala developers as
>>>> Flink could just because they are external projects. It will be a big issue
>>>> for users who have to rewrite their large codebases. Those users should be
>>>> aware of the effort from now on and would better not count on those scala
>>>> extension projects and prepare their migration plan before Flink 2.0.
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>>
>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Marton,
>>>>>
>>>>> You're making a good point, I originally wanted to include already the
>>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>>> more outreach via other channels as well.
>>>>>
>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala
>>>>> API support in a future version of Flink. Your feedback on this topic is
>>>>> very much appreciated.
>>>>>
>>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>>> think that we currently have those in our community. The easiest solution
>>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>>> there's still time for that :)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <
>>>>> balassi.marton@gmail.com> wrote:
>>>>>
>>>>>> Hi Martjin,
>>>>>>
>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>>> poses
>>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>>> 2.12.8
>>>>>> supports) are hanging stale. With that said before we make this move
>>>>>> we
>>>>>> should attempt to understand the userbase affected.
>>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>>> results for scala (admittedly a cursory look at them suggest that
>>>>>> many of
>>>>>> them have to do with missing features in Scala that exist in Java or
>>>>>> Scala
>>>>>> versions). I would love to see some polls on this topic, we could
>>>>>> also use
>>>>>> the Flink twitter handle to ask the community about this.
>>>>>>
>>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>>> This
>>>>>> move would pose a very large effort on them, as they would need to
>>>>>> rewrite
>>>>>> much of their existing code. What are the alternatives in your
>>>>>> opinion,
>>>>>> Martjin?
>>>>>>
>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>>> martijnvisser@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi everyone,
>>>>>> >
>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>>> remove
>>>>>> > Scala API support. Please take a look at
>>>>>> >
>>>>>> >
>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>>> > and provide your feedback.
>>>>>> >
>>>>>> > Best regards,
>>>>>> >
>>>>>> > Martijn
>>>>>> > https://twitter.com/MartijnVisser82
>>>>>> > https://github.com/MartijnVisser
>>>>>> >
>>>>>>
>>>>>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Chesnay Schepler <ch...@apache.org>.
> It's possible that for the sake of the Scala API, we would
occasionally require some changes in the Java API. As long as those
changes are not detrimental to Java users, they should be considered.
That's exactly the model we're trying to get to. Don't fix
scala-specific issues with scala code, but rather on the Java side as
much as possible which could also benefit other JVM languages (e.g.,
Kotlin).
> A question regarding the Flink wrapper: would it be possible to keep
it under the Flink project's umbrella? Or does it need to be a
completely separate structure? I'm not aware of the full organizational
implications of this, I'm afraid.
Technically it can be under the Flink umbrella, but then Flink would
still be (at least for a while) be the bottleneck because we'd have to
review any changes coming in.
That would only improve once several new committers were added to take
care of this project.
(In the end you'd just split Flink and the Scala API _codebases_, but
achieve little else)
> And if that is what it takes to move beyond Scala 2.12.7… This has
been a big pain point for us.
I'm curious what target Scala versions people are currently interested in.
I would've expected that everyone wants to migrate to Scala 3, for which
several wrapper projects around Flink already exist.
On 05/10/2022 12:35, Gaël Renoux wrote:
> Hello everyone,
>
> I've already answered a bit on Twitter, I'll develop my thoughts a bit
> here. For context, my company (DataDome) has a significant codebase on
> Scala Flink (around 110K LOC), having been using it since 2017. I
> myself am an enthusiastic Scala developer (I don't think I'd like
> moving back to Java)
>
> Given that, I think separating the Scala support from Flink is
> actually a good move long term. We would then have a full-Java Flink,
> and a separate Scala wrapper. It would help a lot in solving the
> skills issue: Flink maintainers would no longer need to be fluent in
> Scala, and maintainers of the Scala wrapper would not need a deep
> knowledge of Flink's inner workings, just the API would be sufficient.
> And if that is what it takes to move beyond Scala 2.12.7… This has
> been a big pain point for us.
>
> I'm not too worried about finding contributors for the Scala wrapper.
> Within my company, we have developed additional wrappers and extension
> methods (for parts where we felt the Flink Scala API was
> insufficient), and we've been looking at ways we could contribute
> back. What held us back was our lack of knowledge of the full Flink
> environment (we're only using the Scala Datastream API). I don't think
> we're the only ones in that situation. One major point, though, is
> that Flink developers would not be completely rid of us ;-) It's
> possible that for the sake of the Scala API, we would occasionally
> require some changes in the Java API. As long as those changes are not
> detrimental to Java users, they should be considered.
>
> A question regarding the Flink wrapper: would it be possible to keep
> it under the Flink project's umbrella? Or does it need to be a
> completely separate structure? I'm not aware of the full
> organizational implications of this, I'm afraid.
>
> Finally, the hard part would be the migration to the new version. My
> dream solution would be to have the existing Scala API be entirely
> converted into a Scala wrapper over the Java API. That way, migration
> would be pretty minimal: add a dependency, change the imports for the
> Scala API, and we're done. However, even starting from the existing
> flink4s project, that's still quite a lot of work. So, more
> realistically, I'd settle for at least a partial implementation. We
> would have some broken code that we could fix, but at the very least
> I'd like the basic DataStream functions (process, uid, name…) to be
> available.
>
> Thanks for all the work that went into making Flink what it is!
>
>
> Gaël Renoux - Lead R&D Engineer
> E - gael.renoux@datadome.co
> W - www.datadome.co <http://www.datadome.co>
>
>
>
> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>
> Hi Martin,
>
> Could you please remind what was the conclusion of discussion on
> upgrading Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t -
> I couldn't find any follow-up vote?
>
> If it's acceptable to break binary compatibility by such an
> upgrade, then upgrading to JDK17 before 2.0 will be doable?
>
>
> thanks,
>
> maciek
>
>
> On 04.10.2022 18:21, Martijn Visser wrote:
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java
>> 17 as a prerequisite, we would have to break compatibility
>> already since Scala 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major
>> Flink (2.0) version, would that still impact you a lot? I do
>> imagine that if we get to a Flink 2.0 version there would be more
>> breaking involved anyway. The biggest consequence of deprecating
>> support for Scala in Flink 1.x would be that new APIs would only
>> be available in Java, but since these don't exist yet there would
>> be no refactoring involved. I can imagine that we might change
>> something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen
>> there, I think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko
>> <ya...@goldsky.com> wrote:
>>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm
>> not looking forward to rewriting my codebase, and it's not
>> even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite
>> (https://issues.apache.org/jira/browse/FLINK-15736). Things
>> like switch expressions and records could simplify the
>> migration quite a bit. Would you consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com>
>> wrote:
>>
>> Hi Martijn,
>>
>> Thanks for bringing this up. It is generally a great
>> idea, so +1.
>>
>> Since both scala extension projects mentioned in the FLIP
>> are still very young and I don't think they will attract
>> more scala developers as Flink could just because they
>> are external projects. It will be a big issue for users
>> who have to rewrite their large codebases. Those users
>> should be aware of the effort from now on and would
>> better not count on those scala extension projects and
>> prepare their migration plan before Flink 2.0.
>>
>> Best regards,
>> Jing
>>
>>
>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser
>> <ma...@apache.org> wrote:
>>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to
>> include already the User mailing list to get their
>> feedback but forgot to do so. I'll do some more
>> outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate
>> and remove Scala API support in a future version of
>> Flink. Your feedback on this topic is very much
>> appreciated.
>>
>> Regarding the large Scala codebase for Flink, a
>> potential alternative could be to have a wrapper for
>> all Java APIs that makes them available as Scala
>> APIs. However, this still requires Scala maintainers
>> and I don't think that we currently have those in our
>> community. The easiest solution for them would be to
>> use the Java APIs directly. Yes it would involve
>> work, but we won't actually be able to remove the
>> Scala APIs until Flink 2.0 so there's still time for
>> that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
>> <ba...@gmail.com> wrote:
>>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the
>> sentiment that Scala poses
>> considerable maintenance overhead and key
>> improvements (like 2.13 or 2.12.8
>> supports) are hanging stale. With that said
>> before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does
>> return quite a bit of
>> results for scala (admittedly a cursory look at
>> them suggest that many of
>> them have to do with missing features in Scala
>> that exist in Java or Scala
>> versions). I would love to see some polls on this
>> topic, we could also use
>> the Flink twitter handle to ask the community
>> about this.
>>
>> I am aware of users having large existing Scala
>> codebases for Flink. This
>> move would pose a very large effort on them, as
>> they would need to rewrite
>> much of their existing code. What are the
>> alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
>> <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on
>> FLIP-265 Deprecate and remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Chesnay Schepler <ch...@apache.org>.
> It's possible that for the sake of the Scala API, we would
occasionally require some changes in the Java API. As long as those
changes are not detrimental to Java users, they should be considered.
That's exactly the model we're trying to get to. Don't fix
scala-specific issues with scala code, but rather on the Java side as
much as possible which could also benefit other JVM languages (e.g.,
Kotlin).
> A question regarding the Flink wrapper: would it be possible to keep
it under the Flink project's umbrella? Or does it need to be a
completely separate structure? I'm not aware of the full organizational
implications of this, I'm afraid.
Technically it can be under the Flink umbrella, but then Flink would
still be (at least for a while) be the bottleneck because we'd have to
review any changes coming in.
That would only improve once several new committers were added to take
care of this project.
(In the end you'd just split Flink and the Scala API _codebases_, but
achieve little else)
> And if that is what it takes to move beyond Scala 2.12.7… This has
been a big pain point for us.
I'm curious what target Scala versions people are currently interested in.
I would've expected that everyone wants to migrate to Scala 3, for which
several wrapper projects around Flink already exist.
On 05/10/2022 12:35, Gaël Renoux wrote:
> Hello everyone,
>
> I've already answered a bit on Twitter, I'll develop my thoughts a bit
> here. For context, my company (DataDome) has a significant codebase on
> Scala Flink (around 110K LOC), having been using it since 2017. I
> myself am an enthusiastic Scala developer (I don't think I'd like
> moving back to Java)
>
> Given that, I think separating the Scala support from Flink is
> actually a good move long term. We would then have a full-Java Flink,
> and a separate Scala wrapper. It would help a lot in solving the
> skills issue: Flink maintainers would no longer need to be fluent in
> Scala, and maintainers of the Scala wrapper would not need a deep
> knowledge of Flink's inner workings, just the API would be sufficient.
> And if that is what it takes to move beyond Scala 2.12.7… This has
> been a big pain point for us.
>
> I'm not too worried about finding contributors for the Scala wrapper.
> Within my company, we have developed additional wrappers and extension
> methods (for parts where we felt the Flink Scala API was
> insufficient), and we've been looking at ways we could contribute
> back. What held us back was our lack of knowledge of the full Flink
> environment (we're only using the Scala Datastream API). I don't think
> we're the only ones in that situation. One major point, though, is
> that Flink developers would not be completely rid of us ;-) It's
> possible that for the sake of the Scala API, we would occasionally
> require some changes in the Java API. As long as those changes are not
> detrimental to Java users, they should be considered.
>
> A question regarding the Flink wrapper: would it be possible to keep
> it under the Flink project's umbrella? Or does it need to be a
> completely separate structure? I'm not aware of the full
> organizational implications of this, I'm afraid.
>
> Finally, the hard part would be the migration to the new version. My
> dream solution would be to have the existing Scala API be entirely
> converted into a Scala wrapper over the Java API. That way, migration
> would be pretty minimal: add a dependency, change the imports for the
> Scala API, and we're done. However, even starting from the existing
> flink4s project, that's still quite a lot of work. So, more
> realistically, I'd settle for at least a partial implementation. We
> would have some broken code that we could fix, but at the very least
> I'd like the basic DataStream functions (process, uid, name…) to be
> available.
>
> Thanks for all the work that went into making Flink what it is!
>
>
> Gaël Renoux - Lead R&D Engineer
> E - gael.renoux@datadome.co
> W - www.datadome.co <http://www.datadome.co>
>
>
>
> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
>
> Hi Martin,
>
> Could you please remind what was the conclusion of discussion on
> upgrading Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t -
> I couldn't find any follow-up vote?
>
> If it's acceptable to break binary compatibility by such an
> upgrade, then upgrading to JDK17 before 2.0 will be doable?
>
>
> thanks,
>
> maciek
>
>
> On 04.10.2022 18:21, Martijn Visser wrote:
>> Hi Yaroslav,
>>
>> Thanks for the feedback, that's much appreciated! Regarding Java
>> 17 as a prerequisite, we would have to break compatibility
>> already since Scala 2.12.7 doesn't compile on Java 17 [1].
>>
>> Given that we can only remove Scala APIs with the next major
>> Flink (2.0) version, would that still impact you a lot? I do
>> imagine that if we get to a Flink 2.0 version there would be more
>> breaking involved anyway. The biggest consequence of deprecating
>> support for Scala in Flink 1.x would be that new APIs would only
>> be available in Java, but since these don't exist yet there would
>> be no refactoring involved. I can imagine that we might change
>> something in an existing API, but that would have certain
>> compatibility guarantees already (depending if it's
>> Public/PublicEvolving/Experimental). If a change would happen
>> there, I think it would be smaller refactoring.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-25000
>>
>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko
>> <ya...@goldsky.com> wrote:
>>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm
>> not looking forward to rewriting my codebase, and it's not
>> even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite
>> (https://issues.apache.org/jira/browse/FLINK-15736). Things
>> like switch expressions and records could simplify the
>> migration quite a bit. Would you consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com>
>> wrote:
>>
>> Hi Martijn,
>>
>> Thanks for bringing this up. It is generally a great
>> idea, so +1.
>>
>> Since both scala extension projects mentioned in the FLIP
>> are still very young and I don't think they will attract
>> more scala developers as Flink could just because they
>> are external projects. It will be a big issue for users
>> who have to rewrite their large codebases. Those users
>> should be aware of the effort from now on and would
>> better not count on those scala extension projects and
>> prepare their migration plan before Flink 2.0.
>>
>> Best regards,
>> Jing
>>
>>
>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser
>> <ma...@apache.org> wrote:
>>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to
>> include already the User mailing list to get their
>> feedback but forgot to do so. I'll do some more
>> outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate
>> and remove Scala API support in a future version of
>> Flink. Your feedback on this topic is very much
>> appreciated.
>>
>> Regarding the large Scala codebase for Flink, a
>> potential alternative could be to have a wrapper for
>> all Java APIs that makes them available as Scala
>> APIs. However, this still requires Scala maintainers
>> and I don't think that we currently have those in our
>> community. The easiest solution for them would be to
>> use the Java APIs directly. Yes it would involve
>> work, but we won't actually be able to remove the
>> Scala APIs until Flink 2.0 so there's still time for
>> that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
>> <ba...@gmail.com> wrote:
>>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the
>> sentiment that Scala poses
>> considerable maintenance overhead and key
>> improvements (like 2.13 or 2.12.8
>> supports) are hanging stale. With that said
>> before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does
>> return quite a bit of
>> results for scala (admittedly a cursory look at
>> them suggest that many of
>> them have to do with missing features in Scala
>> that exist in Java or Scala
>> versions). I would love to see some polls on this
>> topic, we could also use
>> the Flink twitter handle to ask the community
>> about this.
>>
>> I am aware of users having large existing Scala
>> codebases for Flink. This
>> move would pose a very large effort on them, as
>> they would need to rewrite
>> much of their existing code. What are the
>> alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
>> <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on
>> FLIP-265 Deprecate and remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Gaël Renoux <ga...@datadome.co.INVALID>.
Hello everyone,
I've already answered a bit on Twitter, I'll develop my thoughts a bit
here. For context, my company (DataDome) has a significant codebase on
Scala Flink (around 110K LOC), having been using it since 2017. I myself am
an enthusiastic Scala developer (I don't think I'd like moving back to
Java)
Given that, I think separating the Scala support from Flink is actually a
good move long term. We would then have a full-Java Flink, and a separate
Scala wrapper. It would help a lot in solving the skills issue: Flink
maintainers would no longer need to be fluent in Scala, and maintainers of
the Scala wrapper would not need a deep knowledge of Flink's inner
workings, just the API would be sufficient. And if that is what it takes to
move beyond Scala 2.12.7… This has been a big pain point for us.
I'm not too worried about finding contributors for the Scala wrapper.
Within my company, we have developed additional wrappers and extension
methods (for parts where we felt the Flink Scala API was insufficient), and
we've been looking at ways we could contribute back. What held us back was
our lack of knowledge of the full Flink environment (we're only using the
Scala Datastream API). I don't think we're the only ones in that situation.
One major point, though, is that Flink developers would not be completely
rid of us ;-) It's possible that for the sake of the Scala API, we would
occasionally require some changes in the Java API. As long as those changes
are not detrimental to Java users, they should be considered.
A question regarding the Flink wrapper: would it be possible to keep it
under the Flink project's umbrella? Or does it need to be a completely
separate structure? I'm not aware of the full organizational implications
of this, I'm afraid.
Finally, the hard part would be the migration to the new version. My dream
solution would be to have the existing Scala API be entirely converted into
a Scala wrapper over the Java API. That way, migration would be pretty
minimal: add a dependency, change the imports for the Scala API, and we're
done. However, even starting from the existing flink4s project, that's
still quite a lot of work. So, more realistically, I'd settle for at least
a partial implementation. We would have some broken code that we could fix,
but at the very least I'd like the basic DataStream functions (process,
uid, name…) to be available.
Thanks for all the work that went into making Flink what it is!
Gaël Renoux - Lead R&D Engineer
E - gael.renoux@datadome.co
W - www.datadome.co
On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
> Hi Martin,
>
> Could you please remind what was the conclusion of discussion on upgrading
> Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
> couldn't find any follow-up vote?
>
> If it's acceptable to break binary compatibility by such an upgrade, then
> upgrading to JDK17 before 2.0 will be doable?
>
>
> thanks,
>
> maciek
>
>
> On 04.10.2022 18:21, Martijn Visser wrote:
>
> Hi Yaroslav,
>
> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
> prerequisite, we would have to break compatibility already since Scala
> 2.12.7 doesn't compile on Java 17 [1].
>
> Given that we can only remove Scala APIs with the next major Flink (2.0)
> version, would that still impact you a lot? I do imagine that if we get to
> a Flink 2.0 version there would be more breaking involved anyway. The
> biggest consequence of deprecating support for Scala in Flink 1.x would be
> that new APIs would only be available in Java, but since these don't exist
> yet there would be no refactoring involved. I can imagine that we might
> change something in an existing API, but that would have certain
> compatibility guarantees already (depending if it's
> Public/PublicEvolving/Experimental). If a change would happen there, I
> think it would be smaller refactoring.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-25000
>
> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm not looking
>> forward to rewriting my codebase, and it's not even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite (
>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>> expressions and records could simplify the migration quite a bit. Would you
>> consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>
>>> Hi Martijn,
>>>
>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>
>>> Since both scala extension projects mentioned in the FLIP are still very
>>> young and I don't think they will attract more scala developers as Flink
>>> could just because they are external projects. It will be a big issue for
>>> users who have to rewrite their large codebases. Those users should be
>>> aware of the effort from now on and would better not count on those scala
>>> extension projects and prepare their migration plan before Flink 2.0.
>>>
>>> Best regards,
>>> Jing
>>>
>>>
>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>>> Hi Marton,
>>>>
>>>> You're making a good point, I originally wanted to include already the
>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>> more outreach via other channels as well.
>>>>
>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>>> support in a future version of Flink. Your feedback on this topic is very
>>>> much appreciated.
>>>>
>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>> think that we currently have those in our community. The easiest solution
>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>> there's still time for that :)
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Martjin,
>>>>>
>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>> poses
>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>> 2.12.8
>>>>> supports) are hanging stale. With that said before we make this move we
>>>>> should attempt to understand the userbase affected.
>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>> results for scala (admittedly a cursory look at them suggest that many
>>>>> of
>>>>> them have to do with missing features in Scala that exist in Java or
>>>>> Scala
>>>>> versions). I would love to see some polls on this topic, we could also
>>>>> use
>>>>> the Flink twitter handle to ask the community about this.
>>>>>
>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>> This
>>>>> move would pose a very large effort on them, as they would need to
>>>>> rewrite
>>>>> much of their existing code. What are the alternatives in your opinion,
>>>>> Martjin?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>> martijnvisser@apache.org>
>>>>> wrote:
>>>>>
>>>>> > Hi everyone,
>>>>> >
>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>> remove
>>>>> > Scala API support. Please take a look at
>>>>> >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>> > and provide your feedback.
>>>>> >
>>>>> > Best regards,
>>>>> >
>>>>> > Martijn
>>>>> > https://twitter.com/MartijnVisser82
>>>>> > https://github.com/MartijnVisser
>>>>> >
>>>>>
>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Gaël Renoux <ga...@datadome.co>.
Hello everyone,
I've already answered a bit on Twitter, I'll develop my thoughts a bit
here. For context, my company (DataDome) has a significant codebase on
Scala Flink (around 110K LOC), having been using it since 2017. I myself am
an enthusiastic Scala developer (I don't think I'd like moving back to
Java)
Given that, I think separating the Scala support from Flink is actually a
good move long term. We would then have a full-Java Flink, and a separate
Scala wrapper. It would help a lot in solving the skills issue: Flink
maintainers would no longer need to be fluent in Scala, and maintainers of
the Scala wrapper would not need a deep knowledge of Flink's inner
workings, just the API would be sufficient. And if that is what it takes to
move beyond Scala 2.12.7… This has been a big pain point for us.
I'm not too worried about finding contributors for the Scala wrapper.
Within my company, we have developed additional wrappers and extension
methods (for parts where we felt the Flink Scala API was insufficient), and
we've been looking at ways we could contribute back. What held us back was
our lack of knowledge of the full Flink environment (we're only using the
Scala Datastream API). I don't think we're the only ones in that situation.
One major point, though, is that Flink developers would not be completely
rid of us ;-) It's possible that for the sake of the Scala API, we would
occasionally require some changes in the Java API. As long as those changes
are not detrimental to Java users, they should be considered.
A question regarding the Flink wrapper: would it be possible to keep it
under the Flink project's umbrella? Or does it need to be a completely
separate structure? I'm not aware of the full organizational implications
of this, I'm afraid.
Finally, the hard part would be the migration to the new version. My dream
solution would be to have the existing Scala API be entirely converted into
a Scala wrapper over the Java API. That way, migration would be pretty
minimal: add a dependency, change the imports for the Scala API, and we're
done. However, even starting from the existing flink4s project, that's
still quite a lot of work. So, more realistically, I'd settle for at least
a partial implementation. We would have some broken code that we could fix,
but at the very least I'd like the basic DataStream functions (process,
uid, name…) to be available.
Thanks for all the work that went into making Flink what it is!
Gaël Renoux - Lead R&D Engineer
E - gael.renoux@datadome.co
W - www.datadome.co
On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <mp...@touk.pl> wrote:
> Hi Martin,
>
> Could you please remind what was the conclusion of discussion on upgrading
> Scala to 2.12.15/16?
> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
> couldn't find any follow-up vote?
>
> If it's acceptable to break binary compatibility by such an upgrade, then
> upgrading to JDK17 before 2.0 will be doable?
>
>
> thanks,
>
> maciek
>
>
> On 04.10.2022 18:21, Martijn Visser wrote:
>
> Hi Yaroslav,
>
> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
> prerequisite, we would have to break compatibility already since Scala
> 2.12.7 doesn't compile on Java 17 [1].
>
> Given that we can only remove Scala APIs with the next major Flink (2.0)
> version, would that still impact you a lot? I do imagine that if we get to
> a Flink 2.0 version there would be more breaking involved anyway. The
> biggest consequence of deprecating support for Scala in Flink 1.x would be
> that new APIs would only be available in Java, but since these don't exist
> yet there would be no refactoring involved. I can imagine that we might
> change something in an existing API, but that would have certain
> compatibility guarantees already (depending if it's
> Public/PublicEvolving/Experimental). If a change would happen there, I
> think it would be smaller refactoring.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-25000
>
> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm not looking
>> forward to rewriting my codebase, and it's not even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite (
>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>> expressions and records could simplify the migration quite a bit. Would you
>> consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>
>>> Hi Martijn,
>>>
>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>
>>> Since both scala extension projects mentioned in the FLIP are still very
>>> young and I don't think they will attract more scala developers as Flink
>>> could just because they are external projects. It will be a big issue for
>>> users who have to rewrite their large codebases. Those users should be
>>> aware of the effort from now on and would better not count on those scala
>>> extension projects and prepare their migration plan before Flink 2.0.
>>>
>>> Best regards,
>>> Jing
>>>
>>>
>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>>> Hi Marton,
>>>>
>>>> You're making a good point, I originally wanted to include already the
>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>> more outreach via other channels as well.
>>>>
>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>>> support in a future version of Flink. Your feedback on this topic is very
>>>> much appreciated.
>>>>
>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>> think that we currently have those in our community. The easiest solution
>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>> there's still time for that :)
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Martjin,
>>>>>
>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>> poses
>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>> 2.12.8
>>>>> supports) are hanging stale. With that said before we make this move we
>>>>> should attempt to understand the userbase affected.
>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>> results for scala (admittedly a cursory look at them suggest that many
>>>>> of
>>>>> them have to do with missing features in Scala that exist in Java or
>>>>> Scala
>>>>> versions). I would love to see some polls on this topic, we could also
>>>>> use
>>>>> the Flink twitter handle to ask the community about this.
>>>>>
>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>> This
>>>>> move would pose a very large effort on them, as they would need to
>>>>> rewrite
>>>>> much of their existing code. What are the alternatives in your opinion,
>>>>> Martjin?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>> martijnvisser@apache.org>
>>>>> wrote:
>>>>>
>>>>> > Hi everyone,
>>>>> >
>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>> remove
>>>>> > Scala API support. Please take a look at
>>>>> >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>> > and provide your feedback.
>>>>> >
>>>>> > Best regards,
>>>>> >
>>>>> > Martijn
>>>>> > https://twitter.com/MartijnVisser82
>>>>> > https://github.com/MartijnVisser
>>>>> >
>>>>>
>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Maciek Próchniak <mp...@touk.pl>.
Hi Martin,
Could you please remind what was the conclusion of discussion on
upgrading Scala to 2.12.15/16?
https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I
couldn't find any follow-up vote?
If it's acceptable to break binary compatibility by such an upgrade,
then upgrading to JDK17 before 2.0 will be doable?
thanks,
maciek
On 04.10.2022 18:21, Martijn Visser wrote:
> Hi Yaroslav,
>
> Thanks for the feedback, that's much appreciated! Regarding Java 17 as
> a prerequisite, we would have to break compatibility already since
> Scala 2.12.7 doesn't compile on Java 17 [1].
>
> Given that we can only remove Scala APIs with the next major Flink
> (2.0) version, would that still impact you a lot? I do imagine that if
> we get to a Flink 2.0 version there would be more breaking involved
> anyway. The biggest consequence of deprecating support for Scala in
> Flink 1.x would be that new APIs would only be available in Java, but
> since these don't exist yet there would be no refactoring involved. I
> can imagine that we might change something in an existing API, but
> that would have certain compatibility guarantees already (depending if
> it's Public/PublicEvolving/Experimental). If a change would happen
> there, I think it would be smaller refactoring.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-25000
>
> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko
> <ya...@goldsky.com> wrote:
>
> Hi Martijn,
>
> As a Scala user, this change would affect me a lot and I'm not
> looking forward to rewriting my codebase, and it's not even a very
> large one :)
>
> I'd like to suggest supporting Java 17 as a prerequisite
> (https://issues.apache.org/jira/browse/FLINK-15736). Things like
> switch expressions and records could simplify the migration
> quite a bit. Would you consider adding it to the FLIP?
>
> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>
> Hi Martijn,
>
> Thanks for bringing this up. It is generally a great idea, so +1.
>
> Since both scala extension projects mentioned in the FLIP are
> still very young and I don't think they will attract more
> scala developers as Flink could just because they are external
> projects. It will be a big issue for users who have to rewrite
> their large codebases. Those users should be aware of the
> effort from now on and would better not count on those scala
> extension projects and prepare their migration plan
> before Flink 2.0.
>
> Best regards,
> Jing
>
>
> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser
> <ma...@apache.org> wrote:
>
> Hi Marton,
>
> You're making a good point, I originally wanted to include
> already the User mailing list to get their feedback but
> forgot to do so. I'll do some more outreach via other
> channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and
> remove Scala API support in a future version of Flink.
> Your feedback on this topic is very much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential
> alternative could be to have a wrapper for all Java APIs
> that makes them available as Scala APIs. However, this
> still requires Scala maintainers and I don't think that we
> currently have those in our community. The easiest
> solution for them would be to use the Java APIs directly.
> Yes it would involve work, but we won't actually be able
> to remove the Scala APIs until Flink 2.0 so there's still
> time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
> <ba...@gmail.com> wrote:
>
> Hi Martjin,
>
> Thanks for compiling the FLIP. I agree with the
> sentiment that Scala poses
> considerable maintenance overhead and key improvements
> (like 2.13 or 2.12.8
> supports) are hanging stale. With that said before we
> make this move we
> should attempt to understand the userbase affected.
> A quick Slack and user mailing list search does return
> quite a bit of
> results for scala (admittedly a cursory look at them
> suggest that many of
> them have to do with missing features in Scala that
> exist in Java or Scala
> versions). I would love to see some polls on this
> topic, we could also use
> the Flink twitter handle to ask the community about this.
>
> I am aware of users having large existing Scala
> codebases for Flink. This
> move would pose a very large effort on them, as they
> would need to rewrite
> much of their existing code. What are the alternatives
> in your opinion,
> Martjin?
>
> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
> <ma...@apache.org>
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a discussion thread on FLIP-265
> Deprecate and remove
> > Scala API support. Please take a look at
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
> > and provide your feedback.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com.INVALID>.
Hi Martijn,
The 2.0 argument makes sense (I agree it's easier to introduce more
breaking changes in one major release), but I think my comment about Java
17 also makes sense in this case: 1) easier to introduce because breaking
changes are possible 2) you'd need to give some syntax sugar as an
alternative after removing Scala.
On Tue, Oct 4, 2022 at 11:21 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Yaroslav,
>
> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
> prerequisite, we would have to break compatibility already since Scala
> 2.12.7 doesn't compile on Java 17 [1].
>
> Given that we can only remove Scala APIs with the next major Flink (2.0)
> version, would that still impact you a lot? I do imagine that if we get to
> a Flink 2.0 version there would be more breaking involved anyway. The
> biggest consequence of deprecating support for Scala in Flink 1.x would be
> that new APIs would only be available in Java, but since these don't exist
> yet there would be no refactoring involved. I can imagine that we might
> change something in an existing API, but that would have certain
> compatibility guarantees already (depending if it's
> Public/PublicEvolving/Experimental). If a change would happen there, I
> think it would be smaller refactoring.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-25000
>
> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hi Martijn,
>>
>> As a Scala user, this change would affect me a lot and I'm not looking
>> forward to rewriting my codebase, and it's not even a very large one :)
>>
>> I'd like to suggest supporting Java 17 as a prerequisite (
>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
>> expressions and records could simplify the migration quite a bit. Would you
>> consider adding it to the FLIP?
>>
>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>>
>>> Hi Martijn,
>>>
>>> Thanks for bringing this up. It is generally a great idea, so +1.
>>>
>>> Since both scala extension projects mentioned in the FLIP are still very
>>> young and I don't think they will attract more scala developers as Flink
>>> could just because they are external projects. It will be a big issue for
>>> users who have to rewrite their large codebases. Those users should be
>>> aware of the effort from now on and would better not count on those scala
>>> extension projects and prepare their migration plan before Flink 2.0.
>>>
>>> Best regards,
>>> Jing
>>>
>>>
>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>>> Hi Marton,
>>>>
>>>> You're making a good point, I originally wanted to include already the
>>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>>> more outreach via other channels as well.
>>>>
>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>>> support in a future version of Flink. Your feedback on this topic is very
>>>> much appreciated.
>>>>
>>>> Regarding the large Scala codebase for Flink, a potential alternative
>>>> could be to have a wrapper for all Java APIs that makes them available as
>>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>>> think that we currently have those in our community. The easiest solution
>>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>>> there's still time for that :)
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Martjin,
>>>>>
>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>>> poses
>>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>>> 2.12.8
>>>>> supports) are hanging stale. With that said before we make this move we
>>>>> should attempt to understand the userbase affected.
>>>>> A quick Slack and user mailing list search does return quite a bit of
>>>>> results for scala (admittedly a cursory look at them suggest that many
>>>>> of
>>>>> them have to do with missing features in Scala that exist in Java or
>>>>> Scala
>>>>> versions). I would love to see some polls on this topic, we could also
>>>>> use
>>>>> the Flink twitter handle to ask the community about this.
>>>>>
>>>>> I am aware of users having large existing Scala codebases for Flink.
>>>>> This
>>>>> move would pose a very large effort on them, as they would need to
>>>>> rewrite
>>>>> much of their existing code. What are the alternatives in your opinion,
>>>>> Martjin?
>>>>>
>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <
>>>>> martijnvisser@apache.org>
>>>>> wrote:
>>>>>
>>>>> > Hi everyone,
>>>>> >
>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>>> remove
>>>>> > Scala API support. Please take a look at
>>>>> >
>>>>> >
>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>>> > and provide your feedback.
>>>>> >
>>>>> > Best regards,
>>>>> >
>>>>> > Martijn
>>>>> > https://twitter.com/MartijnVisser82
>>>>> > https://github.com/MartijnVisser
>>>>> >
>>>>>
>>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi Yaroslav,
Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
prerequisite, we would have to break compatibility already since Scala
2.12.7 doesn't compile on Java 17 [1].
Given that we can only remove Scala APIs with the next major Flink (2.0)
version, would that still impact you a lot? I do imagine that if we get to
a Flink 2.0 version there would be more breaking involved anyway. The
biggest consequence of deprecating support for Scala in Flink 1.x would be
that new APIs would only be available in Java, but since these don't exist
yet there would be no refactoring involved. I can imagine that we might
change something in an existing API, but that would have certain
compatibility guarantees already (depending if it's
Public/PublicEvolving/Experimental). If a change would happen there, I
think it would be smaller refactoring.
Best regards,
Martijn
[1] https://issues.apache.org/jira/browse/FLINK-25000
On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Martijn,
>
> As a Scala user, this change would affect me a lot and I'm not looking
> forward to rewriting my codebase, and it's not even a very large one :)
>
> I'd like to suggest supporting Java 17 as a prerequisite (
> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
> expressions and records could simplify the migration quite a bit. Would you
> consider adding it to the FLIP?
>
> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>
>> Hi Martijn,
>>
>> Thanks for bringing this up. It is generally a great idea, so +1.
>>
>> Since both scala extension projects mentioned in the FLIP are still very
>> young and I don't think they will attract more scala developers as Flink
>> could just because they are external projects. It will be a big issue for
>> users who have to rewrite their large codebases. Those users should be
>> aware of the effort from now on and would better not count on those scala
>> extension projects and prepare their migration plan before Flink 2.0.
>>
>> Best regards,
>> Jing
>>
>>
>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>> support in a future version of Flink. Your feedback on this topic is very
>>> much appreciated.
>>>
>>> Regarding the large Scala codebase for Flink, a potential alternative
>>> could be to have a wrapper for all Java APIs that makes them available as
>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>> think that we currently have those in our community. The easiest solution
>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>> there's still time for that :)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi Martjin,
>>>>
>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>> poses
>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>> 2.12.8
>>>> supports) are hanging stale. With that said before we make this move we
>>>> should attempt to understand the userbase affected.
>>>> A quick Slack and user mailing list search does return quite a bit of
>>>> results for scala (admittedly a cursory look at them suggest that many
>>>> of
>>>> them have to do with missing features in Scala that exist in Java or
>>>> Scala
>>>> versions). I would love to see some polls on this topic, we could also
>>>> use
>>>> the Flink twitter handle to ask the community about this.
>>>>
>>>> I am aware of users having large existing Scala codebases for Flink.
>>>> This
>>>> move would pose a very large effort on them, as they would need to
>>>> rewrite
>>>> much of their existing code. What are the alternatives in your opinion,
>>>> Martjin?
>>>>
>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <martijnvisser@apache.org
>>>> >
>>>> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>> remove
>>>> > Scala API support. Please take a look at
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>> > and provide your feedback.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Martijn
>>>> > https://twitter.com/MartijnVisser82
>>>> > https://github.com/MartijnVisser
>>>> >
>>>>
>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi Yaroslav,
Thanks for the feedback, that's much appreciated! Regarding Java 17 as a
prerequisite, we would have to break compatibility already since Scala
2.12.7 doesn't compile on Java 17 [1].
Given that we can only remove Scala APIs with the next major Flink (2.0)
version, would that still impact you a lot? I do imagine that if we get to
a Flink 2.0 version there would be more breaking involved anyway. The
biggest consequence of deprecating support for Scala in Flink 1.x would be
that new APIs would only be available in Java, but since these don't exist
yet there would be no refactoring involved. I can imagine that we might
change something in an existing API, but that would have certain
compatibility guarantees already (depending if it's
Public/PublicEvolving/Experimental). If a change would happen there, I
think it would be smaller refactoring.
Best regards,
Martijn
[1] https://issues.apache.org/jira/browse/FLINK-25000
On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Martijn,
>
> As a Scala user, this change would affect me a lot and I'm not looking
> forward to rewriting my codebase, and it's not even a very large one :)
>
> I'd like to suggest supporting Java 17 as a prerequisite (
> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
> expressions and records could simplify the migration quite a bit. Would you
> consider adding it to the FLIP?
>
> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
>
>> Hi Martijn,
>>
>> Thanks for bringing this up. It is generally a great idea, so +1.
>>
>> Since both scala extension projects mentioned in the FLIP are still very
>> young and I don't think they will attract more scala developers as Flink
>> could just because they are external projects. It will be a big issue for
>> users who have to rewrite their large codebases. Those users should be
>> aware of the effort from now on and would better not count on those scala
>> extension projects and prepare their migration plan before Flink 2.0.
>>
>> Best regards,
>> Jing
>>
>>
>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>> support in a future version of Flink. Your feedback on this topic is very
>>> much appreciated.
>>>
>>> Regarding the large Scala codebase for Flink, a potential alternative
>>> could be to have a wrapper for all Java APIs that makes them available as
>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>> think that we currently have those in our community. The easiest solution
>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>> there's still time for that :)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi Martjin,
>>>>
>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>> poses
>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>> 2.12.8
>>>> supports) are hanging stale. With that said before we make this move we
>>>> should attempt to understand the userbase affected.
>>>> A quick Slack and user mailing list search does return quite a bit of
>>>> results for scala (admittedly a cursory look at them suggest that many
>>>> of
>>>> them have to do with missing features in Scala that exist in Java or
>>>> Scala
>>>> versions). I would love to see some polls on this topic, we could also
>>>> use
>>>> the Flink twitter handle to ask the community about this.
>>>>
>>>> I am aware of users having large existing Scala codebases for Flink.
>>>> This
>>>> move would pose a very large effort on them, as they would need to
>>>> rewrite
>>>> much of their existing code. What are the alternatives in your opinion,
>>>> Martjin?
>>>>
>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <martijnvisser@apache.org
>>>> >
>>>> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>> remove
>>>> > Scala API support. Please take a look at
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>> > and provide your feedback.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Martijn
>>>> > https://twitter.com/MartijnVisser82
>>>> > https://github.com/MartijnVisser
>>>> >
>>>>
>>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Hi Martijn,
As a Scala user, this change would affect me a lot and I'm not looking
forward to rewriting my codebase, and it's not even a very large one :)
I'd like to suggest supporting Java 17 as a prerequisite (
https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
expressions and records could simplify the migration quite a bit. Would you
consider adding it to the FLIP?
On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
> Hi Martijn,
>
> Thanks for bringing this up. It is generally a great idea, so +1.
>
> Since both scala extension projects mentioned in the FLIP are still very
> young and I don't think they will attract more scala developers as Flink
> could just because they are external projects. It will be a big issue for
> users who have to rewrite their large codebases. Those users should be
> aware of the effort from now on and would better not count on those scala
> extension projects and prepare their migration plan before Flink 2.0.
>
> Best regards,
> Jing
>
>
> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include already the
>> User mailing list to get their feedback but forgot to do so. I'll do some
>> more outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>> support in a future version of Flink. Your feedback on this topic is very
>> much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential alternative
>> could be to have a wrapper for all Java APIs that makes them available as
>> Scala APIs. However, this still requires Scala maintainers and I don't
>> think that we currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would involve work,
>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>> there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>>> Hi Martjin,
>>>
>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>> poses
>>> considerable maintenance overhead and key improvements (like 2.13 or
>>> 2.12.8
>>> supports) are hanging stale. With that said before we make this move we
>>> should attempt to understand the userbase affected.
>>> A quick Slack and user mailing list search does return quite a bit of
>>> results for scala (admittedly a cursory look at them suggest that many of
>>> them have to do with missing features in Scala that exist in Java or
>>> Scala
>>> versions). I would love to see some polls on this topic, we could also
>>> use
>>> the Flink twitter handle to ask the community about this.
>>>
>>> I am aware of users having large existing Scala codebases for Flink. This
>>> move would pose a very large effort on them, as they would need to
>>> rewrite
>>> much of their existing code. What are the alternatives in your opinion,
>>> Martjin?
>>>
>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>> remove
>>> > Scala API support. Please take a look at
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>> > and provide your feedback.
>>> >
>>> > Best regards,
>>> >
>>> > Martijn
>>> > https://twitter.com/MartijnVisser82
>>> > https://github.com/MartijnVisser
>>> >
>>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Yaroslav Tkachenko <ya...@goldsky.com.INVALID>.
Hi Martijn,
As a Scala user, this change would affect me a lot and I'm not looking
forward to rewriting my codebase, and it's not even a very large one :)
I'd like to suggest supporting Java 17 as a prerequisite (
https://issues.apache.org/jira/browse/FLINK-15736). Things like switch
expressions and records could simplify the migration quite a bit. Would you
consider adding it to the FLIP?
On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <ji...@ververica.com> wrote:
> Hi Martijn,
>
> Thanks for bringing this up. It is generally a great idea, so +1.
>
> Since both scala extension projects mentioned in the FLIP are still very
> young and I don't think they will attract more scala developers as Flink
> could just because they are external projects. It will be a big issue for
> users who have to rewrite their large codebases. Those users should be
> aware of the effort from now on and would better not count on those scala
> extension projects and prepare their migration plan before Flink 2.0.
>
> Best regards,
> Jing
>
>
> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include already the
>> User mailing list to get their feedback but forgot to do so. I'll do some
>> more outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>> support in a future version of Flink. Your feedback on this topic is very
>> much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential alternative
>> could be to have a wrapper for all Java APIs that makes them available as
>> Scala APIs. However, this still requires Scala maintainers and I don't
>> think that we currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would involve work,
>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>> there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>>> Hi Martjin,
>>>
>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>> poses
>>> considerable maintenance overhead and key improvements (like 2.13 or
>>> 2.12.8
>>> supports) are hanging stale. With that said before we make this move we
>>> should attempt to understand the userbase affected.
>>> A quick Slack and user mailing list search does return quite a bit of
>>> results for scala (admittedly a cursory look at them suggest that many of
>>> them have to do with missing features in Scala that exist in Java or
>>> Scala
>>> versions). I would love to see some polls on this topic, we could also
>>> use
>>> the Flink twitter handle to ask the community about this.
>>>
>>> I am aware of users having large existing Scala codebases for Flink. This
>>> move would pose a very large effort on them, as they would need to
>>> rewrite
>>> much of their existing code. What are the alternatives in your opinion,
>>> Martjin?
>>>
>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>> remove
>>> > Scala API support. Please take a look at
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>> > and provide your feedback.
>>> >
>>> > Best regards,
>>> >
>>> > Martijn
>>> > https://twitter.com/MartijnVisser82
>>> > https://github.com/MartijnVisser
>>> >
>>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Jing Ge <ji...@ververica.com>.
Hi Martijn,
Thanks for bringing this up. It is generally a great idea, so +1.
Since both scala extension projects mentioned in the FLIP are still very
young and I don't think they will attract more scala developers as Flink
could just because they are external projects. It will be a big issue for
users who have to rewrite their large codebases. Those users should be
aware of the effort from now on and would better not count on those scala
extension projects and prepare their migration plan before Flink 2.0.
Best regards,
Jing
On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Marton,
>
> You're making a good point, I originally wanted to include already the
> User mailing list to get their feedback but forgot to do so. I'll do some
> more outreach via other channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and remove Scala API
> support in a future version of Flink. Your feedback on this topic is very
> much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential alternative
> could be to have a wrapper for all Java APIs that makes them available as
> Scala APIs. However, this still requires Scala maintainers and I don't
> think that we currently have those in our community. The easiest solution
> for them would be to use the Java APIs directly. Yes it would involve work,
> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
> there's still time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
>> considerable maintenance overhead and key improvements (like 2.13 or
>> 2.12.8
>> supports) are hanging stale. With that said before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return quite a bit of
>> results for scala (admittedly a cursory look at them suggest that many of
>> them have to do with missing features in Scala that exist in Java or Scala
>> versions). I would love to see some polls on this topic, we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases for Flink. This
>> move would pose a very large effort on them, as they would need to rewrite
>> much of their existing code. What are the alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>> remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Márton Balassi <ba...@gmail.com>.
Hi Martjin,
After some more careful consideration I am in favor of dropping the Scala
API support in with Flink 2.0 given that we add Java 17 support earlier or
latest at the same time.
Best,
Marton
On Thu, Oct 13, 2022 at 12:01 PM Chesnay Schepler <ch...@apache.org>
wrote:
> Support for records has not been investigated yet. We're still at the
> stage of getting things to run at all on Java 17.
>
> It _may_ be possible, it _may_ not be.
>
> On 13/10/2022 07:39, Salva Alcántara wrote:
>
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written "manually"
> or using Lombok) to Java records without breaking serialization? (assuming
> that those POJOs are used as immutable values, e.g., setters are never
> used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi everyone,
>>
>> Thanks again for all your feedback. It's very much appreciated.
>>
>> My overall feeling is that people are not opposed to the FLIP. There is
>> demand for adding Java 17 support before dropping the Scala APIs. Given
>> that the proposal for actually dropping the Scala APIs would only happen
>> with a Flink 2.0 and Java 17 support would either happen in a new minor
>> version or a new major version (I haven't seen a FLIP or discussion being
>> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
>> would either be there earlier (in a new minor version) or at the same time
>> (with Flink 2.0) when the Scala APIs would be dropped.
>>
>> If there are no more discussion topics, I would move this FLIP to a vote
>> at the beginning of next week.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
>> wrote:
>>
>>> Hi Martijn
>>>
>>> I do not maintain a large production application based on Flink, so it
>>> would not be a problem for me to convert existing implementations to
>>> whatever API.
>>>
>>> I am working in the area of cultural heritage, which is mainly about the
>>> processing of structured (meta)-data (scientific libraries, archives and
>>> museums)
>>> My impression: People without much background/experience with Java
>>> implementations find it easier to get into the functional mindset as
>>> supported in Scala. That's why I think it would be very unfortunate if the
>>> use of Scala in Flink becomes more and more limited or neglected.
>>>
>>> I think using the Java API in Scala is a possible way also in my
>>> environment.
>>>
>>> In the last weeks I tried to port the examples from the "Flink Course"
>>> of Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>>> Scala courses), which are exclusively based on the native Scala API, to the
>>> Java API. This has worked without any problems as far as I can see. So far
>>> I haven't tried any examples based on the Table API or streaming workflows
>>> in batch mode (which would be important for our environment).
>>>
>>> My main trouble: So far I don't know enough about the limitations of
>>> using the Java API in a Scala implementation and what that means. My
>>> current understanding: the limitation is mainly in deriving the type
>>> information in generic APIs with Scala types. For me it would be very
>>> significant and helpful if there would be more information, descriptions
>>> and examples about this topic.
>>>
>>> So far unfortunately I had too little time to deal with a wrapper like
>>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>>> current alternative is probably going to be deprecated in the future (
>>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>>
>>> Günter
>>>
>>>
>>> On 04.10.22 13:58, Martijn Visser wrote:
>>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>> support in a future version of Flink. Your feedback on this topic is very
>>> much appreciated.
>>>
>>> Regarding the large Scala codebase for Flink, a potential alternative
>>> could be to have a wrapper for all Java APIs that makes them available as
>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>> think that we currently have those in our community. The easiest solution
>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>> there's still time for that :)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi Martjin,
>>>>
>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>> poses
>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>> 2.12.8
>>>> supports) are hanging stale. With that said before we make this move we
>>>> should attempt to understand the userbase affected.
>>>> A quick Slack and user mailing list search does return quite a bit of
>>>> results for scala (admittedly a cursory look at them suggest that many
>>>> of
>>>> them have to do with missing features in Scala that exist in Java or
>>>> Scala
>>>> versions). I would love to see some polls on this topic, we could also
>>>> use
>>>> the Flink twitter handle to ask the community about this.
>>>>
>>>> I am aware of users having large existing Scala codebases for Flink.
>>>> This
>>>> move would pose a very large effort on them, as they would need to
>>>> rewrite
>>>> much of their existing code. What are the alternatives in your opinion,
>>>> Martjin?
>>>>
>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <martijnvisser@apache.org
>>>> >
>>>> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>> remove
>>>> > Scala API support. Please take a look at
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>> > and provide your feedback.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Martijn
>>>> > https://twitter.com/MartijnVisser82
>>>> > https://github.com/MartijnVisser
>>>> >
>>>>
>>> --
>>> Günter Hipler
>>> University library Leipzig
>>>
>>>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Márton Balassi <ba...@gmail.com>.
Hi Martjin,
After some more careful consideration I am in favor of dropping the Scala
API support in with Flink 2.0 given that we add Java 17 support earlier or
latest at the same time.
Best,
Marton
On Thu, Oct 13, 2022 at 12:01 PM Chesnay Schepler <ch...@apache.org>
wrote:
> Support for records has not been investigated yet. We're still at the
> stage of getting things to run at all on Java 17.
>
> It _may_ be possible, it _may_ not be.
>
> On 13/10/2022 07:39, Salva Alcántara wrote:
>
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written "manually"
> or using Lombok) to Java records without breaking serialization? (assuming
> that those POJOs are used as immutable values, e.g., setters are never
> used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi everyone,
>>
>> Thanks again for all your feedback. It's very much appreciated.
>>
>> My overall feeling is that people are not opposed to the FLIP. There is
>> demand for adding Java 17 support before dropping the Scala APIs. Given
>> that the proposal for actually dropping the Scala APIs would only happen
>> with a Flink 2.0 and Java 17 support would either happen in a new minor
>> version or a new major version (I haven't seen a FLIP or discussion being
>> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
>> would either be there earlier (in a new minor version) or at the same time
>> (with Flink 2.0) when the Scala APIs would be dropped.
>>
>> If there are no more discussion topics, I would move this FLIP to a vote
>> at the beginning of next week.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
>> wrote:
>>
>>> Hi Martijn
>>>
>>> I do not maintain a large production application based on Flink, so it
>>> would not be a problem for me to convert existing implementations to
>>> whatever API.
>>>
>>> I am working in the area of cultural heritage, which is mainly about the
>>> processing of structured (meta)-data (scientific libraries, archives and
>>> museums)
>>> My impression: People without much background/experience with Java
>>> implementations find it easier to get into the functional mindset as
>>> supported in Scala. That's why I think it would be very unfortunate if the
>>> use of Scala in Flink becomes more and more limited or neglected.
>>>
>>> I think using the Java API in Scala is a possible way also in my
>>> environment.
>>>
>>> In the last weeks I tried to port the examples from the "Flink Course"
>>> of Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>>> Scala courses), which are exclusively based on the native Scala API, to the
>>> Java API. This has worked without any problems as far as I can see. So far
>>> I haven't tried any examples based on the Table API or streaming workflows
>>> in batch mode (which would be important for our environment).
>>>
>>> My main trouble: So far I don't know enough about the limitations of
>>> using the Java API in a Scala implementation and what that means. My
>>> current understanding: the limitation is mainly in deriving the type
>>> information in generic APIs with Scala types. For me it would be very
>>> significant and helpful if there would be more information, descriptions
>>> and examples about this topic.
>>>
>>> So far unfortunately I had too little time to deal with a wrapper like
>>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>>> current alternative is probably going to be deprecated in the future (
>>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>>
>>> Günter
>>>
>>>
>>> On 04.10.22 13:58, Martijn Visser wrote:
>>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>> support in a future version of Flink. Your feedback on this topic is very
>>> much appreciated.
>>>
>>> Regarding the large Scala codebase for Flink, a potential alternative
>>> could be to have a wrapper for all Java APIs that makes them available as
>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>> think that we currently have those in our community. The easiest solution
>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>> there's still time for that :)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi Martjin,
>>>>
>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>>> poses
>>>> considerable maintenance overhead and key improvements (like 2.13 or
>>>> 2.12.8
>>>> supports) are hanging stale. With that said before we make this move we
>>>> should attempt to understand the userbase affected.
>>>> A quick Slack and user mailing list search does return quite a bit of
>>>> results for scala (admittedly a cursory look at them suggest that many
>>>> of
>>>> them have to do with missing features in Scala that exist in Java or
>>>> Scala
>>>> versions). I would love to see some polls on this topic, we could also
>>>> use
>>>> the Flink twitter handle to ask the community about this.
>>>>
>>>> I am aware of users having large existing Scala codebases for Flink.
>>>> This
>>>> move would pose a very large effort on them, as they would need to
>>>> rewrite
>>>> much of their existing code. What are the alternatives in your opinion,
>>>> Martjin?
>>>>
>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <martijnvisser@apache.org
>>>> >
>>>> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>>> remove
>>>> > Scala API support. Please take a look at
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>>> > and provide your feedback.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Martijn
>>>> > https://twitter.com/MartijnVisser82
>>>> > https://github.com/MartijnVisser
>>>> >
>>>>
>>> --
>>> Günter Hipler
>>> University library Leipzig
>>>
>>>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Chesnay Schepler <ch...@apache.org>.
Support for records has not been investigated yet. We're still at the
stage of getting things to run at all on Java 17.
It _may_ be possible, it _may_ not be.
On 13/10/2022 07:39, Salva Alcántara wrote:
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written
> "manually" or using Lombok) to Java records without breaking
> serialization? (assuming that those POJOs are used as immutable
> values, e.g., setters are never used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser
> <ma...@apache.org> wrote:
>
> Hi everyone,
>
> Thanks again for all your feedback. It's very much appreciated.
>
> My overall feeling is that people are not opposed to the FLIP.
> There is demand for adding Java 17 support before dropping the
> Scala APIs. Given that the proposal for actually dropping the
> Scala APIs would only happen with a Flink 2.0 and Java 17 support
> would either happen in a new minor version or a new major version
> (I haven't seen a FLIP or discussion being opened adding Java 17
> support, only on deprecating Java 8), Java 17 support would either
> be there earlier (in a new minor version) or at the same time
> (with Flink 2.0) when the Scala APIs would be dropped.
>
> If there are no more discussion topics, I would move this FLIP to
> a vote at the beginning of next week.
>
> Best regards,
>
> Martijn
>
> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists
> <gu...@bluewin.ch> wrote:
>
> Hi Martijn
>
> I do not maintain a large production application based on
> Flink, so it would not be a problem for me to convert existing
> implementations to whatever API.
>
> I am working in the area of cultural heritage, which is mainly
> about the processing of structured (meta)-data (scientific
> libraries, archives and museums)
> My impression: People without much background/experience with
> Java implementations find it easier to get into the functional
> mindset as supported in Scala. That's why I think it would be
> very unfortunate if the use of Scala in Flink becomes more and
> more limited or neglected.
>
> I think using the Java API in Scala is a possible way also in
> my environment.
>
> In the last weeks I tried to port the examples from the "Flink
> Course" of Daniel Ciorcilan (https://rockthejvm.com/p/flink -
> he mainly offers Scala courses), which are exclusively based
> on the native Scala API, to the Java API. This has worked
> without any problems as far as I can see. So far I haven't
> tried any examples based on the Table API or streaming
> workflows in batch mode (which would be important for our
> environment).
>
> My main trouble: So far I don't know enough about the
> limitations of using the Java API in a Scala implementation
> and what that means. My current understanding: the limitation
> is mainly in deriving the type information in generic APIs
> with Scala types. For me it would be very significant and
> helpful if there would be more information, descriptions and
> examples about this topic.
>
> So far unfortunately I had too little time to deal with a
> wrapper like flink-scala-api
> (https://github.com/findify/flink-scala-api ) and the current
> alternative is probably going to be deprecated in the future
> (https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808
> )
>
> Günter
>
>
> On 04.10.22 13:58, Martijn Visser wrote:
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include
>> already the User mailing list to get their feedback but
>> forgot to do so. I'll do some more outreach via other
>> channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove
>> Scala API support in a future version of Flink. Your feedback
>> on this topic is very much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential
>> alternative could be to have a wrapper for all Java APIs that
>> makes them available as Scala APIs. However, this still
>> requires Scala maintainers and I don't think that we
>> currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would
>> involve work, but we won't actually be able to remove the
>> Scala APIs until Flink 2.0 so there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
>> <ba...@gmail.com> wrote:
>>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment
>> that Scala poses
>> considerable maintenance overhead and key improvements
>> (like 2.13 or 2.12.8
>> supports) are hanging stale. With that said before we
>> make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return
>> quite a bit of
>> results for scala (admittedly a cursory look at them
>> suggest that many of
>> them have to do with missing features in Scala that exist
>> in Java or Scala
>> versions). I would love to see some polls on this topic,
>> we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases
>> for Flink. This
>> move would pose a very large effort on them, as they
>> would need to rewrite
>> much of their existing code. What are the alternatives in
>> your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
>> <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265
>> Deprecate and remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
> --
> Günter Hipler
> University library Leipzig
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Chesnay Schepler <ch...@apache.org>.
Support for records has not been investigated yet. We're still at the
stage of getting things to run at all on Java 17.
It _may_ be possible, it _may_ not be.
On 13/10/2022 07:39, Salva Alcántara wrote:
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written
> "manually" or using Lombok) to Java records without breaking
> serialization? (assuming that those POJOs are used as immutable
> values, e.g., setters are never used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser
> <ma...@apache.org> wrote:
>
> Hi everyone,
>
> Thanks again for all your feedback. It's very much appreciated.
>
> My overall feeling is that people are not opposed to the FLIP.
> There is demand for adding Java 17 support before dropping the
> Scala APIs. Given that the proposal for actually dropping the
> Scala APIs would only happen with a Flink 2.0 and Java 17 support
> would either happen in a new minor version or a new major version
> (I haven't seen a FLIP or discussion being opened adding Java 17
> support, only on deprecating Java 8), Java 17 support would either
> be there earlier (in a new minor version) or at the same time
> (with Flink 2.0) when the Scala APIs would be dropped.
>
> If there are no more discussion topics, I would move this FLIP to
> a vote at the beginning of next week.
>
> Best regards,
>
> Martijn
>
> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists
> <gu...@bluewin.ch> wrote:
>
> Hi Martijn
>
> I do not maintain a large production application based on
> Flink, so it would not be a problem for me to convert existing
> implementations to whatever API.
>
> I am working in the area of cultural heritage, which is mainly
> about the processing of structured (meta)-data (scientific
> libraries, archives and museums)
> My impression: People without much background/experience with
> Java implementations find it easier to get into the functional
> mindset as supported in Scala. That's why I think it would be
> very unfortunate if the use of Scala in Flink becomes more and
> more limited or neglected.
>
> I think using the Java API in Scala is a possible way also in
> my environment.
>
> In the last weeks I tried to port the examples from the "Flink
> Course" of Daniel Ciorcilan (https://rockthejvm.com/p/flink -
> he mainly offers Scala courses), which are exclusively based
> on the native Scala API, to the Java API. This has worked
> without any problems as far as I can see. So far I haven't
> tried any examples based on the Table API or streaming
> workflows in batch mode (which would be important for our
> environment).
>
> My main trouble: So far I don't know enough about the
> limitations of using the Java API in a Scala implementation
> and what that means. My current understanding: the limitation
> is mainly in deriving the type information in generic APIs
> with Scala types. For me it would be very significant and
> helpful if there would be more information, descriptions and
> examples about this topic.
>
> So far unfortunately I had too little time to deal with a
> wrapper like flink-scala-api
> (https://github.com/findify/flink-scala-api ) and the current
> alternative is probably going to be deprecated in the future
> (https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808
> )
>
> Günter
>
>
> On 04.10.22 13:58, Martijn Visser wrote:
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include
>> already the User mailing list to get their feedback but
>> forgot to do so. I'll do some more outreach via other
>> channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove
>> Scala API support in a future version of Flink. Your feedback
>> on this topic is very much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential
>> alternative could be to have a wrapper for all Java APIs that
>> makes them available as Scala APIs. However, this still
>> requires Scala maintainers and I don't think that we
>> currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would
>> involve work, but we won't actually be able to remove the
>> Scala APIs until Flink 2.0 so there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
>> <ba...@gmail.com> wrote:
>>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment
>> that Scala poses
>> considerable maintenance overhead and key improvements
>> (like 2.13 or 2.12.8
>> supports) are hanging stale. With that said before we
>> make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return
>> quite a bit of
>> results for scala (admittedly a cursory look at them
>> suggest that many of
>> them have to do with missing features in Scala that exist
>> in Java or Scala
>> versions). I would love to see some polls on this topic,
>> we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases
>> for Flink. This
>> move would pose a very large effort on them, as they
>> would need to rewrite
>> much of their existing code. What are the alternatives in
>> your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
>> <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265
>> Deprecate and remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
> --
> Günter Hipler
> University library Leipzig
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Salva Alcántara <sa...@gmail.com>.
Hi Martijn,
Maybe a bit of an off-topic, but regarding Java 17 support, will it be
possible to replace POJOs with Java records in existing applications?
In a project I maintain we use Lombok a lot, but with Java records we would
probably stop using it (or significantly reduce its usage).
Will there be a way to promote existing POJOs (either written "manually" or
using Lombok) to Java records without breaking serialization? (assuming
that those POJOs are used as immutable values, e.g., setters are never
used).
Regards,
Salva
On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi everyone,
>
> Thanks again for all your feedback. It's very much appreciated.
>
> My overall feeling is that people are not opposed to the FLIP. There is
> demand for adding Java 17 support before dropping the Scala APIs. Given
> that the proposal for actually dropping the Scala APIs would only happen
> with a Flink 2.0 and Java 17 support would either happen in a new minor
> version or a new major version (I haven't seen a FLIP or discussion being
> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
> would either be there earlier (in a new minor version) or at the same time
> (with Flink 2.0) when the Scala APIs would be dropped.
>
> If there are no more discussion topics, I would move this FLIP to a vote
> at the beginning of next week.
>
> Best regards,
>
> Martijn
>
> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
> wrote:
>
>> Hi Martijn
>>
>> I do not maintain a large production application based on Flink, so it
>> would not be a problem for me to convert existing implementations to
>> whatever API.
>>
>> I am working in the area of cultural heritage, which is mainly about the
>> processing of structured (meta)-data (scientific libraries, archives and
>> museums)
>> My impression: People without much background/experience with Java
>> implementations find it easier to get into the functional mindset as
>> supported in Scala. That's why I think it would be very unfortunate if the
>> use of Scala in Flink becomes more and more limited or neglected.
>>
>> I think using the Java API in Scala is a possible way also in my
>> environment.
>>
>> In the last weeks I tried to port the examples from the "Flink Course" of
>> Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>> Scala courses), which are exclusively based on the native Scala API, to the
>> Java API. This has worked without any problems as far as I can see. So far
>> I haven't tried any examples based on the Table API or streaming workflows
>> in batch mode (which would be important for our environment).
>>
>> My main trouble: So far I don't know enough about the limitations of
>> using the Java API in a Scala implementation and what that means. My
>> current understanding: the limitation is mainly in deriving the type
>> information in generic APIs with Scala types. For me it would be very
>> significant and helpful if there would be more information, descriptions
>> and examples about this topic.
>>
>> So far unfortunately I had too little time to deal with a wrapper like
>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>> current alternative is probably going to be deprecated in the future (
>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>
>> Günter
>>
>>
>> On 04.10.22 13:58, Martijn Visser wrote:
>>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include already the
>> User mailing list to get their feedback but forgot to do so. I'll do some
>> more outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>> support in a future version of Flink. Your feedback on this topic is very
>> much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential alternative
>> could be to have a wrapper for all Java APIs that makes them available as
>> Scala APIs. However, this still requires Scala maintainers and I don't
>> think that we currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would involve work,
>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>> there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>>> Hi Martjin,
>>>
>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>> poses
>>> considerable maintenance overhead and key improvements (like 2.13 or
>>> 2.12.8
>>> supports) are hanging stale. With that said before we make this move we
>>> should attempt to understand the userbase affected.
>>> A quick Slack and user mailing list search does return quite a bit of
>>> results for scala (admittedly a cursory look at them suggest that many of
>>> them have to do with missing features in Scala that exist in Java or
>>> Scala
>>> versions). I would love to see some polls on this topic, we could also
>>> use
>>> the Flink twitter handle to ask the community about this.
>>>
>>> I am aware of users having large existing Scala codebases for Flink. This
>>> move would pose a very large effort on them, as they would need to
>>> rewrite
>>> much of their existing code. What are the alternatives in your opinion,
>>> Martjin?
>>>
>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>> remove
>>> > Scala API support. Please take a look at
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>> > and provide your feedback.
>>> >
>>> > Best regards,
>>> >
>>> > Martijn
>>> > https://twitter.com/MartijnVisser82
>>> > https://github.com/MartijnVisser
>>> >
>>>
>> --
>> Günter Hipler
>> University library Leipzig
>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Salva Alcántara <sa...@gmail.com>.
Hi Martijn,
Maybe a bit of an off-topic, but regarding Java 17 support, will it be
possible to replace POJOs with Java records in existing applications?
In a project I maintain we use Lombok a lot, but with Java records we would
probably stop using it (or significantly reduce its usage).
Will there be a way to promote existing POJOs (either written "manually" or
using Lombok) to Java records without breaking serialization? (assuming
that those POJOs are used as immutable values, e.g., setters are never
used).
Regards,
Salva
On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi everyone,
>
> Thanks again for all your feedback. It's very much appreciated.
>
> My overall feeling is that people are not opposed to the FLIP. There is
> demand for adding Java 17 support before dropping the Scala APIs. Given
> that the proposal for actually dropping the Scala APIs would only happen
> with a Flink 2.0 and Java 17 support would either happen in a new minor
> version or a new major version (I haven't seen a FLIP or discussion being
> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
> would either be there earlier (in a new minor version) or at the same time
> (with Flink 2.0) when the Scala APIs would be dropped.
>
> If there are no more discussion topics, I would move this FLIP to a vote
> at the beginning of next week.
>
> Best regards,
>
> Martijn
>
> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
> wrote:
>
>> Hi Martijn
>>
>> I do not maintain a large production application based on Flink, so it
>> would not be a problem for me to convert existing implementations to
>> whatever API.
>>
>> I am working in the area of cultural heritage, which is mainly about the
>> processing of structured (meta)-data (scientific libraries, archives and
>> museums)
>> My impression: People without much background/experience with Java
>> implementations find it easier to get into the functional mindset as
>> supported in Scala. That's why I think it would be very unfortunate if the
>> use of Scala in Flink becomes more and more limited or neglected.
>>
>> I think using the Java API in Scala is a possible way also in my
>> environment.
>>
>> In the last weeks I tried to port the examples from the "Flink Course" of
>> Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>> Scala courses), which are exclusively based on the native Scala API, to the
>> Java API. This has worked without any problems as far as I can see. So far
>> I haven't tried any examples based on the Table API or streaming workflows
>> in batch mode (which would be important for our environment).
>>
>> My main trouble: So far I don't know enough about the limitations of
>> using the Java API in a Scala implementation and what that means. My
>> current understanding: the limitation is mainly in deriving the type
>> information in generic APIs with Scala types. For me it would be very
>> significant and helpful if there would be more information, descriptions
>> and examples about this topic.
>>
>> So far unfortunately I had too little time to deal with a wrapper like
>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>> current alternative is probably going to be deprecated in the future (
>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>
>> Günter
>>
>>
>> On 04.10.22 13:58, Martijn Visser wrote:
>>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include already the
>> User mailing list to get their feedback but forgot to do so. I'll do some
>> more outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>> support in a future version of Flink. Your feedback on this topic is very
>> much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential alternative
>> could be to have a wrapper for all Java APIs that makes them available as
>> Scala APIs. However, this still requires Scala maintainers and I don't
>> think that we currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would involve work,
>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>> there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>>> Hi Martjin,
>>>
>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>> poses
>>> considerable maintenance overhead and key improvements (like 2.13 or
>>> 2.12.8
>>> supports) are hanging stale. With that said before we make this move we
>>> should attempt to understand the userbase affected.
>>> A quick Slack and user mailing list search does return quite a bit of
>>> results for scala (admittedly a cursory look at them suggest that many of
>>> them have to do with missing features in Scala that exist in Java or
>>> Scala
>>> versions). I would love to see some polls on this topic, we could also
>>> use
>>> the Flink twitter handle to ask the community about this.
>>>
>>> I am aware of users having large existing Scala codebases for Flink. This
>>> move would pose a very large effort on them, as they would need to
>>> rewrite
>>> much of their existing code. What are the alternatives in your opinion,
>>> Martjin?
>>>
>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>>> remove
>>> > Scala API support. Please take a look at
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>>> > and provide your feedback.
>>> >
>>> > Best regards,
>>> >
>>> > Martijn
>>> > https://twitter.com/MartijnVisser82
>>> > https://github.com/MartijnVisser
>>> >
>>>
>> --
>> Günter Hipler
>> University library Leipzig
>>
>>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi everyone,
Thanks again for all your feedback. It's very much appreciated.
My overall feeling is that people are not opposed to the FLIP. There is
demand for adding Java 17 support before dropping the Scala APIs. Given
that the proposal for actually dropping the Scala APIs would only happen
with a Flink 2.0 and Java 17 support would either happen in a new minor
version or a new major version (I haven't seen a FLIP or discussion being
opened adding Java 17 support, only on deprecating Java 8), Java 17 support
would either be there earlier (in a new minor version) or at the same time
(with Flink 2.0) when the Scala APIs would be dropped.
If there are no more discussion topics, I would move this FLIP to a vote at
the beginning of next week.
Best regards,
Martijn
On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
wrote:
> Hi Martijn
>
> I do not maintain a large production application based on Flink, so it
> would not be a problem for me to convert existing implementations to
> whatever API.
>
> I am working in the area of cultural heritage, which is mainly about the
> processing of structured (meta)-data (scientific libraries, archives and
> museums)
> My impression: People without much background/experience with Java
> implementations find it easier to get into the functional mindset as
> supported in Scala. That's why I think it would be very unfortunate if the
> use of Scala in Flink becomes more and more limited or neglected.
>
> I think using the Java API in Scala is a possible way also in my
> environment.
>
> In the last weeks I tried to port the examples from the "Flink Course" of
> Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers Scala
> courses), which are exclusively based on the native Scala API, to the Java
> API. This has worked without any problems as far as I can see. So far I
> haven't tried any examples based on the Table API or streaming workflows in
> batch mode (which would be important for our environment).
>
> My main trouble: So far I don't know enough about the limitations of using
> the Java API in a Scala implementation and what that means. My current
> understanding: the limitation is mainly in deriving the type information in
> generic APIs with Scala types. For me it would be very significant and
> helpful if there would be more information, descriptions and examples about
> this topic.
>
> So far unfortunately I had too little time to deal with a wrapper like
> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
> current alternative is probably going to be deprecated in the future (
> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>
> Günter
>
>
> On 04.10.22 13:58, Martijn Visser wrote:
>
> Hi Marton,
>
> You're making a good point, I originally wanted to include already the
> User mailing list to get their feedback but forgot to do so. I'll do some
> more outreach via other channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and remove Scala API
> support in a future version of Flink. Your feedback on this topic is very
> much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential alternative
> could be to have a wrapper for all Java APIs that makes them available as
> Scala APIs. However, this still requires Scala maintainers and I don't
> think that we currently have those in our community. The easiest solution
> for them would be to use the Java APIs directly. Yes it would involve work,
> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
> there's still time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
>> considerable maintenance overhead and key improvements (like 2.13 or
>> 2.12.8
>> supports) are hanging stale. With that said before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return quite a bit of
>> results for scala (admittedly a cursory look at them suggest that many of
>> them have to do with missing features in Scala that exist in Java or Scala
>> versions). I would love to see some polls on this topic, we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases for Flink. This
>> move would pose a very large effort on them, as they would need to rewrite
>> much of their existing code. What are the alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>> remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
> --
> Günter Hipler
> University library Leipzig
>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Martijn Visser <ma...@apache.org>.
Hi everyone,
Thanks again for all your feedback. It's very much appreciated.
My overall feeling is that people are not opposed to the FLIP. There is
demand for adding Java 17 support before dropping the Scala APIs. Given
that the proposal for actually dropping the Scala APIs would only happen
with a Flink 2.0 and Java 17 support would either happen in a new minor
version or a new major version (I haven't seen a FLIP or discussion being
opened adding Java 17 support, only on deprecating Java 8), Java 17 support
would either be there earlier (in a new minor version) or at the same time
(with Flink 2.0) when the Scala APIs would be dropped.
If there are no more discussion topics, I would move this FLIP to a vote at
the beginning of next week.
Best regards,
Martijn
On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists <gu...@bluewin.ch>
wrote:
> Hi Martijn
>
> I do not maintain a large production application based on Flink, so it
> would not be a problem for me to convert existing implementations to
> whatever API.
>
> I am working in the area of cultural heritage, which is mainly about the
> processing of structured (meta)-data (scientific libraries, archives and
> museums)
> My impression: People without much background/experience with Java
> implementations find it easier to get into the functional mindset as
> supported in Scala. That's why I think it would be very unfortunate if the
> use of Scala in Flink becomes more and more limited or neglected.
>
> I think using the Java API in Scala is a possible way also in my
> environment.
>
> In the last weeks I tried to port the examples from the "Flink Course" of
> Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers Scala
> courses), which are exclusively based on the native Scala API, to the Java
> API. This has worked without any problems as far as I can see. So far I
> haven't tried any examples based on the Table API or streaming workflows in
> batch mode (which would be important for our environment).
>
> My main trouble: So far I don't know enough about the limitations of using
> the Java API in a Scala implementation and what that means. My current
> understanding: the limitation is mainly in deriving the type information in
> generic APIs with Scala types. For me it would be very significant and
> helpful if there would be more information, descriptions and examples about
> this topic.
>
> So far unfortunately I had too little time to deal with a wrapper like
> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
> current alternative is probably going to be deprecated in the future (
> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>
> Günter
>
>
> On 04.10.22 13:58, Martijn Visser wrote:
>
> Hi Marton,
>
> You're making a good point, I originally wanted to include already the
> User mailing list to get their feedback but forgot to do so. I'll do some
> more outreach via other channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and remove Scala API
> support in a future version of Flink. Your feedback on this topic is very
> much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential alternative
> could be to have a wrapper for all Java APIs that makes them available as
> Scala APIs. However, this still requires Scala maintainers and I don't
> think that we currently have those in our community. The easiest solution
> for them would be to use the Java APIs directly. Yes it would involve work,
> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
> there's still time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
>> considerable maintenance overhead and key improvements (like 2.13 or
>> 2.12.8
>> supports) are hanging stale. With that said before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return quite a bit of
>> results for scala (admittedly a cursory look at them suggest that many of
>> them have to do with missing features in Scala that exist in Java or Scala
>> versions). I would love to see some polls on this topic, we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases for Flink. This
>> move would pose a very large effort on them, as they would need to rewrite
>> much of their existing code. What are the alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>> remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
> --
> Günter Hipler
> University library Leipzig
>
>
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by "guenterh.lists" <gu...@bluewin.ch>.
Hi Martijn
I do not maintain a large production application based on Flink, so it
would not be a problem for me to convert existing implementations to
whatever API.
I am working in the area of cultural heritage, which is mainly about the
processing of structured (meta)-data (scientific libraries, archives and
museums)
My impression: People without much background/experience with Java
implementations find it easier to get into the functional mindset as
supported in Scala. That's why I think it would be very unfortunate if
the use of Scala in Flink becomes more and more limited or neglected.
I think using the Java API in Scala is a possible way also in my
environment.
In the last weeks I tried to port the examples from the "Flink Course"
of Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
Scala courses), which are exclusively based on the native Scala API, to
the Java API. This has worked without any problems as far as I can see.
So far I haven't tried any examples based on the Table API or streaming
workflows in batch mode (which would be important for our environment).
My main trouble: So far I don't know enough about the limitations of
using the Java API in a Scala implementation and what that means. My
current understanding: the limitation is mainly in deriving the type
information in generic APIs with Scala types. For me it would be very
significant and helpful if there would be more information, descriptions
and examples about this topic.
So far unfortunately I had too little time to deal with a wrapper like
flink-scala-api (https://github.com/findify/flink-scala-api ) and the
current alternative is probably going to be deprecated in the future
(https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
Günter
On 04.10.22 13:58, Martijn Visser wrote:
> Hi Marton,
>
> You're making a good point, I originally wanted to include already the
> User mailing list to get their feedback but forgot to do so. I'll do
> some more outreach via other channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and remove Scala
> API support in a future version of Flink. Your feedback on this topic
> is very much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential alternative
> could be to have a wrapper for all Java APIs that makes them available
> as Scala APIs. However, this still requires Scala maintainers and I
> don't think that we currently have those in our community. The easiest
> solution for them would be to use the Java APIs directly. Yes it would
> involve work, but we won't actually be able to remove the Scala APIs
> until Flink 2.0 so there's still time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
> <ba...@gmail.com> wrote:
>
> Hi Martjin,
>
> Thanks for compiling the FLIP. I agree with the sentiment that
> Scala poses
> considerable maintenance overhead and key improvements (like 2.13
> or 2.12.8
> supports) are hanging stale. With that said before we make this
> move we
> should attempt to understand the userbase affected.
> A quick Slack and user mailing list search does return quite a bit of
> results for scala (admittedly a cursory look at them suggest that
> many of
> them have to do with missing features in Scala that exist in Java
> or Scala
> versions). I would love to see some polls on this topic, we could
> also use
> the Flink twitter handle to ask the community about this.
>
> I am aware of users having large existing Scala codebases for
> Flink. This
> move would pose a very large effort on them, as they would need to
> rewrite
> much of their existing code. What are the alternatives in your
> opinion,
> Martjin?
>
> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
> <ma...@apache.org>
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a discussion thread on FLIP-265 Deprecate
> and remove
> > Scala API support. Please take a look at
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
> > and provide your feedback.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
>
--
Günter Hipler
University library Leipzig
Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support
Posted by Jing Ge <ji...@ververica.com>.
Hi Martijn,
Thanks for bringing this up. It is generally a great idea, so +1.
Since both scala extension projects mentioned in the FLIP are still very
young and I don't think they will attract more scala developers as Flink
could just because they are external projects. It will be a big issue for
users who have to rewrite their large codebases. Those users should be
aware of the effort from now on and would better not count on those scala
extension projects and prepare their migration plan before Flink 2.0.
Best regards,
Jing
On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Marton,
>
> You're making a good point, I originally wanted to include already the
> User mailing list to get their feedback but forgot to do so. I'll do some
> more outreach via other channels as well.
>
> @Users of Flink, I've made a proposal to deprecate and remove Scala API
> support in a future version of Flink. Your feedback on this topic is very
> much appreciated.
>
> Regarding the large Scala codebase for Flink, a potential alternative
> could be to have a wrapper for all Java APIs that makes them available as
> Scala APIs. However, this still requires Scala maintainers and I don't
> think that we currently have those in our community. The easiest solution
> for them would be to use the Java APIs directly. Yes it would involve work,
> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
> there's still time for that :)
>
> Best regards,
>
> Martijn
>
> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
>> Hi Martjin,
>>
>> Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
>> considerable maintenance overhead and key improvements (like 2.13 or
>> 2.12.8
>> supports) are hanging stale. With that said before we make this move we
>> should attempt to understand the userbase affected.
>> A quick Slack and user mailing list search does return quite a bit of
>> results for scala (admittedly a cursory look at them suggest that many of
>> them have to do with missing features in Scala that exist in Java or Scala
>> versions). I would love to see some polls on this topic, we could also use
>> the Flink twitter handle to ask the community about this.
>>
>> I am aware of users having large existing Scala codebases for Flink. This
>> move would pose a very large effort on them, as they would need to rewrite
>> much of their existing code. What are the alternatives in your opinion,
>> Martjin?
>>
>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I would like to open a discussion thread on FLIP-265 Deprecate and
>> remove
>> > Scala API support. Please take a look at
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
>> > and provide your feedback.
>> >
>> > Best regards,
>> >
>> > Martijn
>> > https://twitter.com/MartijnVisser82
>> > https://github.com/MartijnVisser
>> >
>>
>