You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Salva Alcántara <sa...@gmail.com> on 2022/05/06 16:36:38 UTC
Practical guidance with Scala and Flink >= 1.15
I've always used Scala in the context of Flink. Now that Flink 1.15 has
become Scala-free, I wonder what is the best (most practical) route for me
moving forward. These are my options:
1. Keep using Scala 2.12 for the years to come (and upgrade to newer
versions when the community has come up with something). How long is Flink
expected to support Scala 2.12?
2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly (without
any Scala-specific wrapper/API). How problematic will that be, especially
regarding type information & scala-specific serializers? I hate those
"returns" (type hints) in the Java API...
3. Switch to Java, at least for the time being...
To be clear, I have a strong preference for Scala over Java, but I'm trying
to look at the "grand scheme of things" here, and be pragmatic. I guess I'm
not alone here, and that many people are indeed evaluating the same pros &
cons. Any feedback will be much appreciated.
Thanks in advance!
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Roman Grebennikov <gr...@dfdx.me>.
As a yet another attempt to make Flink work with scala 2.13/3.x, we went further and and cross-built a forked version of Flink's Scala API: https://github.com/findify/flink-scala-api <https://github.com/findify/flink-scala-api.>
Check the github repo for details, but if you can afford re-bootstrapping your job state from scratch (as there is no savepoint compatibility, see the repo readme for details), then migration is quite a straightforward process. Main features:
* Cross-built for Scala 2.12, 2.13 and 3.x
* 100% replica of the 1.15 Scala API, with deprecated methods removed.
* ClosureCleaner from Spark 3.x with 2.13 and java 17 support.
* Agnostic to the serialization format, but can use flink-adt to support scala more natively (with sealed traits and scala3 typeclasses instead of scary macro).
It's still an experimental version which is not yet well tested, but I don't see any possible dangers in it:
* we're using flink-adt exclusively for a lot of Flink jobs in Findify for almost a year with no issues so far.
* Flink's Scala API is a thin wrapper on top of Java API, so there is nothing to break.
The only not well-tested thing is scala3 derivation, which is a compile-time process. So if you have compilation errors due to flink-adt failing for your case classes, you're welcome to submit a bug report on github.
with best regards,
Roman Grebennikov | grv@dfdx.me
On Wed, May 11, 2022, at 07:38, Ran Tao wrote:
> Hi, guys. I posted a jdk11 & jdk17 issue [FLINK-27549] <https://issues.apache.org/jira/browse/FLINK-27549> recently which involved upgrading scala [2] of current discussion.
>
> It shows that the current flink project is not a completed or pure jdk11 version.(same problem with higher version). because the generated scala bytecode <=1.8 can't generate target-11 or higher bytecode version. If scala upgrade to 2.13 will fix this problem.
>
> Hope it helps~
>
> [1] https://lists.apache.org/list?dev@flink.apache.org:lte=1M:Ran%20Tao
>
> Martijn Visser <ma...@apache.org> 于2022年5月11日周三 15:10写道：
>> Hi Matthias,
>>
>> Given the current state of Scala support in the Flink community (there is a major lack in Scala maintainers), it is my personal opinion that we should consider deprecating the current Scala APIs and replace those with new Scala APIs, which are 'just' wrappers for the Java API. This definitely needs a FLIP and a discussion, where I can explain the situation more. There's nothing concrete for this yet though. There have been some discussions on this topic going on in the ticket for adding Scala 2.13 support [1]
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555
>>
>> On Tue, 10 May 2022 at 16:16, Schwalbe Matthias <Ma...@viseca.ch> wrote:
>>> … just for my understanding____
>>> __ __
>>> From the announcements I only got that scala remains only a dependency in the JARs that relate to the Scala API.____
>>> I never read about plans to drop the Scala API altogether … is that the case??____
>>> That would be very unfortunate …____
>>> __ __
>>> What is the state of the affair?____
>>> __ __
>>> Best regards____
>>> __ __
>>> Thias____
>>> __ __
>>> __ __
>>> __ __
>>> *From:* Martijn Visser <ma...@apache.org>
>>> *Sent:* Monday, May 9, 2022 2:38 PM
>>> *To:* Robert Metzger <me...@gmail.com>
>>> *Cc:* Salva Alcántara <sa...@gmail.com>; user <us...@flink.apache.org>
>>> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15____
>>> __ __
>>> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠____
>>> __ __
>>> Hi Salva,____
>>> __ __
>>> Like Robert said, I don't expect that we will be able to drop support for Scala 2.12 anytime soon. I do think that we should have a discussion in the Flink community about providing Scala APIs. My opinion is that we are probably better off to deprecate the current Scala APIs (keeping it internal as we still have a big piece of Scala internally) and only offer Java APIs. The Flink community lacks real Scala maintainers. I think Seth's blog is pretty spot-on on this too [1].____
>>> __ __
>>> Best regards,____
>>> __ __
>>> Martijn Visser____
>>> https://twitter.com/MartijnVisser82____
>>> https://github.com/MartijnVisser____
>>> __ __
>>> [1] https://flink.apache.org/2022/02/22/scala-free.html____
>>> __ __
>>> On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:____
>>>> Hi Salva,____
>>>> my somewhat wild guess (because I'm not very involved with the Scala development on Flink): I would stick with option 1 for now. It should be easier now for the Flink community to support Scala versions past 2.12 (because we don't need to worry about scala 2.12+ support for Flink's internal dependencies such as akka).____
>>>> An argument against supporting newer Scala versions is that I'm not aware of anybody currently working on Flink with Scala in general.____
>>>> __ __
>>>> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com> wrote:____
>>>>> I've always used Scala in the context of Flink. Now that Flink 1.15 has become Scala-free, I wonder what is the best (most practical) route for me moving forward. These are my options:____
>>>>> __ __
>>>>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer versions when the community has come up with something). How long is Flink expected to support Scala 2.12?____
>>>>> __ __
>>>>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly (without any Scala-specific wrapper/API). How problematic will that be, especially regarding type information & scala-specific serializers? I hate those "returns" (type hints) in the Java API...____
>>>>> __ __
>>>>> 3. Switch to Java, at least for the time being...____
>>>>> __ __
>>>>> To be clear, I have a strong preference for Scala over Java, but I'm trying to look at the "grand scheme of things" here, and be pragmatic. I guess I'm not alone here, and that many people are indeed evaluating the same pros & cons. Any feedback will be much appreciated.____
>>>>> __ __
>>>>> Thanks in advance!____
>>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
>>>
>>> This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
>
>
> --
> Best,
> Ran Tao
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Ran Tao <ch...@gmail.com>.
Hi, guys. I posted a jdk11 & jdk17 issue [FLINK-27549]
<https://issues.apache.org/jira/browse/FLINK-27549> recently which involved
upgrading scala [2] of current discussion.
It shows that the current flink project is not a completed or pure jdk11
version.(same problem with higher version). because the generated scala
bytecode <=1.8 can't generate target-11 or higher bytecode version. If
scala upgrade to 2.13 will fix this problem.
Hope it helps~
[1] https://lists.apache.org/list?dev@flink.apache.org:lte=1M:Ran%20Tao
Martijn Visser <ma...@apache.org> 于2022年5月11日周三 15:10写道：
> Hi Matthias,
>
> Given the current state of Scala support in the Flink community (there is
> a major lack in Scala maintainers), it is my personal opinion that we
> should consider deprecating the current Scala APIs and replace those with
> new Scala APIs, which are 'just' wrappers for the Java API. This definitely
> needs a FLIP and a discussion, where I can explain the situation more.
> There's nothing concrete for this yet though. There have been some
> discussions on this topic going on in the ticket for adding Scala 2.13
> support [1]
>
> Best regards,
>
> Martijn
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555
>
> On Tue, 10 May 2022 at 16:16, Schwalbe Matthias <
> Matthias.Schwalbe@viseca.ch> wrote:
>
>> … just for my understanding
>>
>>
>>
>> From the announcements I only got that scala remains only a dependency in
>> the JARs that relate to the Scala API.
>>
>> I never read about plans to drop the Scala API altogether … is that the
>> case??
>>
>> That would be very unfortunate …
>>
>>
>>
>> What is the state of the affair?
>>
>>
>>
>> Best regards
>>
>>
>>
>> Thias
>>
>>
>>
>>
>>
>>
>>
>> *From:* Martijn Visser <ma...@apache.org>
>> *Sent:* Monday, May 9, 2022 2:38 PM
>> *To:* Robert Metzger <me...@gmail.com>
>> *Cc:* Salva Alcántara <sa...@gmail.com>; user <
>> user@flink.apache.org>
>> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15
>>
>>
>>
>> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>>
>>
>>
>> Hi Salva,
>>
>>
>>
>> Like Robert said, I don't expect that we will be able to drop support for
>> Scala 2.12 anytime soon. I do think that we should have a discussion in the
>> Flink community about providing Scala APIs. My opinion is that we are
>> probably better off to deprecate the current Scala APIs (keeping it
>> internal as we still have a big piece of Scala internally) and only offer
>> Java APIs. The Flink community lacks real Scala maintainers. I think Seth's
>> blog is pretty spot-on on this too [1].
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Martijn Visser
>>
>> https://twitter.com/MartijnVisser82
>>
>> https://github.com/MartijnVisser
>>
>>
>>
>> [1] https://flink.apache.org/2022/02/22/scala-free.html
>>
>>
>>
>> On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:
>>
>> Hi Salva,
>>
>> my somewhat wild guess (because I'm not very involved with the Scala
>> development on Flink): I would stick with option 1 for now. It should be
>> easier now for the Flink community to support Scala versions past 2.12
>> (because we don't need to worry about scala 2.12+ support for Flink's
>> internal dependencies such as akka).
>>
>> An argument against supporting newer Scala versions is that I'm not aware
>> of anybody currently working on Flink with Scala in general.
>>
>>
>>
>> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
>> wrote:
>>
>> I've always used Scala in the context of Flink. Now that Flink 1.15 has
>> become Scala-free, I wonder what is the best (most practical) route for me
>> moving forward. These are my options:
>>
>>
>>
>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
>> versions when the community has come up with something). How long is Flink
>> expected to support Scala 2.12?
>>
>>
>>
>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly
>> (without any Scala-specific wrapper/API). How problematic will that be,
>> especially regarding type information & scala-specific serializers? I hate
>> those "returns" (type hints) in the Java API...
>>
>>
>>
>> 3. Switch to Java, at least for the time being...
>>
>>
>>
>> To be clear, I have a strong preference for Scala over Java, but I'm
>> trying to look at the "grand scheme of things" here, and be pragmatic. I
>> guess I'm not alone here, and that many people are indeed evaluating the
>> same pros & cons. Any feedback will be much appreciated.
>>
>>
>>
>> Thanks in advance!
>>
>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
>> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
>> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
>> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
>> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
>> dieser Informationen ist streng verboten.
>>
>> This message is intended only for the named recipient and may contain
>> confidential or privileged information. As the confidentiality of email
>> communication cannot be guaranteed, we do not accept any responsibility for
>> the confidentiality and the intactness of this message. If you have
>> received it in error, please advise the sender by return e-mail and delete
>> this message and any attachments. Any unauthorised use or dissemination of
>> this information is strictly prohibited.
>>
>
--
Best,
Ran Tao
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Salva Alcántara <sa...@gmail.com>.
Sorry, I forgot the link of the repo:
https://github.com/salvalcantara/flink-scala
Regards,
Salva
On Wed, May 11, 2022 at 9:32 AM Salva Alcántara <sa...@gmail.com>
wrote:
> Thanks Martijn, my conclusion so far is that Java is a safe bet.
>
> In the meantime, a friend and I have spent some time trying to make
> `flink-scala` work with Flink 1.15 and Scala 2.13. We partly followed the
> discussions in [1] (FLINK-13414) to fix all the compilations errors. Note
> that this is just a few-hour (very naive) experiment, but might be useful
> as a starting point.
>
> Thanks,
>
> Salva
>
> On Wed, May 11, 2022 at 9:10 AM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Matthias,
>>
>> Given the current state of Scala support in the Flink community (there is
>> a major lack in Scala maintainers), it is my personal opinion that we
>> should consider deprecating the current Scala APIs and replace those with
>> new Scala APIs, which are 'just' wrappers for the Java API. This definitely
>> needs a FLIP and a discussion, where I can explain the situation more.
>> There's nothing concrete for this yet though. There have been some
>> discussions on this topic going on in the ticket for adding Scala 2.13
>> support [1]
>>
>> Best regards,
>>
>> Martijn
>>
>> [1]
>> https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555
>>
>> On Tue, 10 May 2022 at 16:16, Schwalbe Matthias <
>> Matthias.Schwalbe@viseca.ch> wrote:
>>
>>> … just for my understanding
>>>
>>>
>>>
>>> From the announcements I only got that scala remains only a dependency
>>> in the JARs that relate to the Scala API.
>>>
>>> I never read about plans to drop the Scala API altogether … is that the
>>> case??
>>>
>>> That would be very unfortunate …
>>>
>>>
>>>
>>> What is the state of the affair?
>>>
>>>
>>>
>>> Best regards
>>>
>>>
>>>
>>> Thias
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Martijn Visser <ma...@apache.org>
>>> *Sent:* Monday, May 9, 2022 2:38 PM
>>> *To:* Robert Metzger <me...@gmail.com>
>>> *Cc:* Salva Alcántara <sa...@gmail.com>; user <
>>> user@flink.apache.org>
>>> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15
>>>
>>>
>>>
>>> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>>>
>>>
>>>
>>> Hi Salva,
>>>
>>>
>>>
>>> Like Robert said, I don't expect that we will be able to drop support
>>> for Scala 2.12 anytime soon. I do think that we should have a discussion in
>>> the Flink community about providing Scala APIs. My opinion is that we are
>>> probably better off to deprecate the current Scala APIs (keeping it
>>> internal as we still have a big piece of Scala internally) and only offer
>>> Java APIs. The Flink community lacks real Scala maintainers. I think Seth's
>>> blog is pretty spot-on on this too [1].
>>>
>>>
>>>
>>> Best regards,
>>>
>>>
>>>
>>> Martijn Visser
>>>
>>> https://twitter.com/MartijnVisser82
>>>
>>> https://github.com/MartijnVisser
>>>
>>>
>>>
>>> [1] https://flink.apache.org/2022/02/22/scala-free.html
>>>
>>>
>>>
>>> On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:
>>>
>>> Hi Salva,
>>>
>>> my somewhat wild guess (because I'm not very involved with the Scala
>>> development on Flink): I would stick with option 1 for now. It should be
>>> easier now for the Flink community to support Scala versions past 2.12
>>> (because we don't need to worry about scala 2.12+ support for Flink's
>>> internal dependencies such as akka).
>>>
>>> An argument against supporting newer Scala versions is that I'm not
>>> aware of anybody currently working on Flink with Scala in general.
>>>
>>>
>>>
>>> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
>>> wrote:
>>>
>>> I've always used Scala in the context of Flink. Now that Flink 1.15 has
>>> become Scala-free, I wonder what is the best (most practical) route for me
>>> moving forward. These are my options:
>>>
>>>
>>>
>>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
>>> versions when the community has come up with something). How long is Flink
>>> expected to support Scala 2.12?
>>>
>>>
>>>
>>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly
>>> (without any Scala-specific wrapper/API). How problematic will that be,
>>> especially regarding type information & scala-specific serializers? I hate
>>> those "returns" (type hints) in the Java API...
>>>
>>>
>>>
>>> 3. Switch to Java, at least for the time being...
>>>
>>>
>>>
>>> To be clear, I have a strong preference for Scala over Java, but I'm
>>> trying to look at the "grand scheme of things" here, and be pragmatic. I
>>> guess I'm not alone here, and that many people are indeed evaluating the
>>> same pros & cons. Any feedback will be much appreciated.
>>>
>>>
>>>
>>> Thanks in advance!
>>>
>>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
>>> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
>>> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
>>> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
>>> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
>>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>>> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
>>> dieser Informationen ist streng verboten.
>>>
>>> This message is intended only for the named recipient and may contain
>>> confidential or privileged information. As the confidentiality of email
>>> communication cannot be guaranteed, we do not accept any responsibility for
>>> the confidentiality and the intactness of this message. If you have
>>> received it in error, please advise the sender by return e-mail and delete
>>> this message and any attachments. Any unauthorised use or dissemination of
>>> this information is strictly prohibited.
>>>
>>
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Salva Alcántara <sa...@gmail.com>.
Thanks Martijn, my conclusion so far is that Java is a safe bet.
In the meantime, a friend and I have spent some time trying to make
`flink-scala` work with Flink 1.15 and Scala 2.13. We partly followed the
discussions in [1] (FLINK-13414) to fix all the compilations errors. Note
that this is just a few-hour (very naive) experiment, but might be useful
as a starting point.
Thanks,
Salva
On Wed, May 11, 2022 at 9:10 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Matthias,
>
> Given the current state of Scala support in the Flink community (there is
> a major lack in Scala maintainers), it is my personal opinion that we
> should consider deprecating the current Scala APIs and replace those with
> new Scala APIs, which are 'just' wrappers for the Java API. This definitely
> needs a FLIP and a discussion, where I can explain the situation more.
> There's nothing concrete for this yet though. There have been some
> discussions on this topic going on in the ticket for adding Scala 2.13
> support [1]
>
> Best regards,
>
> Martijn
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555
>
> On Tue, 10 May 2022 at 16:16, Schwalbe Matthias <
> Matthias.Schwalbe@viseca.ch> wrote:
>
>> … just for my understanding
>>
>>
>>
>> From the announcements I only got that scala remains only a dependency in
>> the JARs that relate to the Scala API.
>>
>> I never read about plans to drop the Scala API altogether … is that the
>> case??
>>
>> That would be very unfortunate …
>>
>>
>>
>> What is the state of the affair?
>>
>>
>>
>> Best regards
>>
>>
>>
>> Thias
>>
>>
>>
>>
>>
>>
>>
>> *From:* Martijn Visser <ma...@apache.org>
>> *Sent:* Monday, May 9, 2022 2:38 PM
>> *To:* Robert Metzger <me...@gmail.com>
>> *Cc:* Salva Alcántara <sa...@gmail.com>; user <
>> user@flink.apache.org>
>> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15
>>
>>
>>
>> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>>
>>
>>
>> Hi Salva,
>>
>>
>>
>> Like Robert said, I don't expect that we will be able to drop support for
>> Scala 2.12 anytime soon. I do think that we should have a discussion in the
>> Flink community about providing Scala APIs. My opinion is that we are
>> probably better off to deprecate the current Scala APIs (keeping it
>> internal as we still have a big piece of Scala internally) and only offer
>> Java APIs. The Flink community lacks real Scala maintainers. I think Seth's
>> blog is pretty spot-on on this too [1].
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Martijn Visser
>>
>> https://twitter.com/MartijnVisser82
>>
>> https://github.com/MartijnVisser
>>
>>
>>
>> [1] https://flink.apache.org/2022/02/22/scala-free.html
>>
>>
>>
>> On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:
>>
>> Hi Salva,
>>
>> my somewhat wild guess (because I'm not very involved with the Scala
>> development on Flink): I would stick with option 1 for now. It should be
>> easier now for the Flink community to support Scala versions past 2.12
>> (because we don't need to worry about scala 2.12+ support for Flink's
>> internal dependencies such as akka).
>>
>> An argument against supporting newer Scala versions is that I'm not aware
>> of anybody currently working on Flink with Scala in general.
>>
>>
>>
>> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
>> wrote:
>>
>> I've always used Scala in the context of Flink. Now that Flink 1.15 has
>> become Scala-free, I wonder what is the best (most practical) route for me
>> moving forward. These are my options:
>>
>>
>>
>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
>> versions when the community has come up with something). How long is Flink
>> expected to support Scala 2.12?
>>
>>
>>
>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly
>> (without any Scala-specific wrapper/API). How problematic will that be,
>> especially regarding type information & scala-specific serializers? I hate
>> those "returns" (type hints) in the Java API...
>>
>>
>>
>> 3. Switch to Java, at least for the time being...
>>
>>
>>
>> To be clear, I have a strong preference for Scala over Java, but I'm
>> trying to look at the "grand scheme of things" here, and be pragmatic. I
>> guess I'm not alone here, and that many people are indeed evaluating the
>> same pros & cons. Any feedback will be much appreciated.
>>
>>
>>
>> Thanks in advance!
>>
>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
>> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
>> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
>> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
>> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
>> dieser Informationen ist streng verboten.
>>
>> This message is intended only for the named recipient and may contain
>> confidential or privileged information. As the confidentiality of email
>> communication cannot be guaranteed, we do not accept any responsibility for
>> the confidentiality and the intactness of this message. If you have
>> received it in error, please advise the sender by return e-mail and delete
>> this message and any attachments. Any unauthorised use or dissemination of
>> this information is strictly prohibited.
>>
>
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Martijn Visser <ma...@apache.org>.
Hi Matthias,
Given the current state of Scala support in the Flink community (there is a
major lack in Scala maintainers), it is my personal opinion that we should
consider deprecating the current Scala APIs and replace those with new
Scala APIs, which are 'just' wrappers for the Java API. This definitely
needs a FLIP and a discussion, where I can explain the situation more.
There's nothing concrete for this yet though. There have been some
discussions on this topic going on in the ticket for adding Scala 2.13
support [1]
Best regards,
Martijn
[1]
https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555
On Tue, 10 May 2022 at 16:16, Schwalbe Matthias <Ma...@viseca.ch>
wrote:
> … just for my understanding
>
>
>
> From the announcements I only got that scala remains only a dependency in
> the JARs that relate to the Scala API.
>
> I never read about plans to drop the Scala API altogether … is that the
> case??
>
> That would be very unfortunate …
>
>
>
> What is the state of the affair?
>
>
>
> Best regards
>
>
>
> Thias
>
>
>
>
>
>
>
> *From:* Martijn Visser <ma...@apache.org>
> *Sent:* Monday, May 9, 2022 2:38 PM
> *To:* Robert Metzger <me...@gmail.com>
> *Cc:* Salva Alcántara <sa...@gmail.com>; user <
> user@flink.apache.org>
> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15
>
>
>
> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>
>
>
> Hi Salva,
>
>
>
> Like Robert said, I don't expect that we will be able to drop support for
> Scala 2.12 anytime soon. I do think that we should have a discussion in the
> Flink community about providing Scala APIs. My opinion is that we are
> probably better off to deprecate the current Scala APIs (keeping it
> internal as we still have a big piece of Scala internally) and only offer
> Java APIs. The Flink community lacks real Scala maintainers. I think Seth's
> blog is pretty spot-on on this too [1].
>
>
>
> Best regards,
>
>
>
> Martijn Visser
>
> https://twitter.com/MartijnVisser82
>
> https://github.com/MartijnVisser
>
>
>
> [1] https://flink.apache.org/2022/02/22/scala-free.html
>
>
>
> On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:
>
> Hi Salva,
>
> my somewhat wild guess (because I'm not very involved with the Scala
> development on Flink): I would stick with option 1 for now. It should be
> easier now for the Flink community to support Scala versions past 2.12
> (because we don't need to worry about scala 2.12+ support for Flink's
> internal dependencies such as akka).
>
> An argument against supporting newer Scala versions is that I'm not aware
> of anybody currently working on Flink with Scala in general.
>
>
>
> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
> wrote:
>
> I've always used Scala in the context of Flink. Now that Flink 1.15 has
> become Scala-free, I wonder what is the best (most practical) route for me
> moving forward. These are my options:
>
>
>
> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
> versions when the community has come up with something). How long is Flink
> expected to support Scala 2.12?
>
>
>
> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly (without
> any Scala-specific wrapper/API). How problematic will that be, especially
> regarding type information & scala-specific serializers? I hate those
> "returns" (type hints) in the Java API...
>
>
>
> 3. Switch to Java, at least for the time being...
>
>
>
> To be clear, I have a strong preference for Scala over Java, but I'm
> trying to look at the "grand scheme of things" here, and be pragmatic. I
> guess I'm not alone here, and that many people are indeed evaluating the
> same pros & cons. Any feedback will be much appreciated.
>
>
>
> Thanks in advance!
>
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
RE: Practical guidance with Scala and Flink >= 1.15
Posted by Schwalbe Matthias <Ma...@viseca.ch>.
… just for my understanding
From the announcements I only got that scala remains only a dependency in the JARs that relate to the Scala API.
I never read about plans to drop the Scala API altogether … is that the case??
That would be very unfortunate …
What is the state of the affair?
Best regards
Thias
From: Martijn Visser <ma...@apache.org>
Sent: Monday, May 9, 2022 2:38 PM
To: Robert Metzger <me...@gmail.com>
Cc: Salva Alcántara <sa...@gmail.com>; user <us...@flink.apache.org>
Subject: Re: Practical guidance with Scala and Flink >= 1.15
⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠
Hi Salva,
Like Robert said, I don't expect that we will be able to drop support for Scala 2.12 anytime soon. I do think that we should have a discussion in the Flink community about providing Scala APIs. My opinion is that we are probably better off to deprecate the current Scala APIs (keeping it internal as we still have a big piece of Scala internally) and only offer Java APIs. The Flink community lacks real Scala maintainers. I think Seth's blog is pretty spot-on on this too [1].
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
[1] https://flink.apache.org/2022/02/22/scala-free.html
On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com>> wrote:
Hi Salva,
my somewhat wild guess (because I'm not very involved with the Scala development on Flink): I would stick with option 1 for now. It should be easier now for the Flink community to support Scala versions past 2.12 (because we don't need to worry about scala 2.12+ support for Flink's internal dependencies such as akka).
An argument against supporting newer Scala versions is that I'm not aware of anybody currently working on Flink with Scala in general.
On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>> wrote:
I've always used Scala in the context of Flink. Now that Flink 1.15 has become Scala-free, I wonder what is the best (most practical) route for me moving forward. These are my options:
1. Keep using Scala 2.12 for the years to come (and upgrade to newer versions when the community has come up with something). How long is Flink expected to support Scala 2.12?
2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly (without any Scala-specific wrapper/API). How problematic will that be, especially regarding type information & scala-specific serializers? I hate those "returns" (type hints) in the Java API...
3. Switch to Java, at least for the time being...
To be clear, I have a strong preference for Scala over Java, but I'm trying to look at the "grand scheme of things" here, and be pragmatic. I guess I'm not alone here, and that many people are indeed evaluating the same pros & cons. Any feedback will be much appreciated.
Thanks in advance!
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Martijn Visser <ma...@apache.org>.
Hi Salva,
Like Robert said, I don't expect that we will be able to drop support for
Scala 2.12 anytime soon. I do think that we should have a discussion in the
Flink community about providing Scala APIs. My opinion is that we are
probably better off to deprecate the current Scala APIs (keeping it
internal as we still have a big piece of Scala internally) and only offer
Java APIs. The Flink community lacks real Scala maintainers. I think Seth's
blog is pretty spot-on on this too [1].
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
[1] https://flink.apache.org/2022/02/22/scala-free.html
On Mon, 9 May 2022 at 12:24, Robert Metzger <me...@gmail.com> wrote:
> Hi Salva,
> my somewhat wild guess (because I'm not very involved with the Scala
> development on Flink): I would stick with option 1 for now. It should be
> easier now for the Flink community to support Scala versions past 2.12
> (because we don't need to worry about scala 2.12+ support for Flink's
> internal dependencies such as akka).
> An argument against supporting newer Scala versions is that I'm not aware
> of anybody currently working on Flink with Scala in general.
>
> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
> wrote:
>
>> I've always used Scala in the context of Flink. Now that Flink 1.15 has
>> become Scala-free, I wonder what is the best (most practical) route for me
>> moving forward. These are my options:
>>
>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
>> versions when the community has come up with something). How long is Flink
>> expected to support Scala 2.12?
>>
>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly
>> (without any Scala-specific wrapper/API). How problematic will that be,
>> especially regarding type information & scala-specific serializers? I hate
>> those "returns" (type hints) in the Java API...
>>
>> 3. Switch to Java, at least for the time being...
>>
>> To be clear, I have a strong preference for Scala over Java, but I'm
>> trying to look at the "grand scheme of things" here, and be pragmatic. I
>> guess I'm not alone here, and that many people are indeed evaluating the
>> same pros & cons. Any feedback will be much appreciated.
>>
>> Thanks in advance!
>>
>
Re: Practical guidance with Scala and Flink >= 1.15
Posted by Robert Metzger <me...@gmail.com>.
Hi Salva,
my somewhat wild guess (because I'm not very involved with the Scala
development on Flink): I would stick with option 1 for now. It should be
easier now for the Flink community to support Scala versions past 2.12
(because we don't need to worry about scala 2.12+ support for Flink's
internal dependencies such as akka).
An argument against supporting newer Scala versions is that I'm not aware
of anybody currently working on Flink with Scala in general.
On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <sa...@gmail.com>
wrote:
> I've always used Scala in the context of Flink. Now that Flink 1.15 has
> become Scala-free, I wonder what is the best (most practical) route for me
> moving forward. These are my options:
>
> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer
> versions when the community has come up with something). How long is Flink
> expected to support Scala 2.12?
>
> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly (without
> any Scala-specific wrapper/API). How problematic will that be, especially
> regarding type information & scala-specific serializers? I hate those
> "returns" (type hints) in the Java API...
>
> 3. Switch to Java, at least for the time being...
>
> To be clear, I have a strong preference for Scala over Java, but I'm
> trying to look at the "grand scheme of things" here, and be pragmatic. I
> guess I'm not alone here, and that many people are indeed evaluating the
> same pros & cons. Any feedback will be much appreciated.
>
> Thanks in advance!
>