You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Matthias Berndt <ma...@ttmzero.com> on 2024/01/20 18:34:25 UTC

Kafka-Streams-Scala for Scala 3

Hey there,

I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
Currently, the build system is set up such that kafka-streams-scala
and core (i. e. kafka itself) are compiled with the same Scala
compiler versions. This is not an optimal situation because it means
that a Scala 3 release of kafka-streams-scala cannot happen
independently of kafka itself. I think this should be changed

The production codebase of scala-streams-kafka actually compiles just
fine on Scala 3.3.1 with two lines of trivial syntax changes. The
problem is with the tests. These use the `EmbeddedKafkaCluster` class,
which means that kafka is pulled into the classpath, potentially
leading to binary compatibility issues.
I can see several approaches to fixing this:

1. Run the kafka-streams-scala tests using the compatible version of
:core if one is available. Currently, this means that everything can
be tested (test kafka-streams-scala_2.12 using core_2.12,
kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
using core_2.13, as these should be compatible), but when a new
scala-library version is released that is no longer compatible with
2.13, we won't be able to test that.
2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
running the test cluster in a separate JVM or perhaps even a
container.

I'd be willing to get my hands dirty working on this, but before I
start I'd like to get some feedback from the Kafka team regarding the
approaches outlined above.

All the best
Matthias Berndt

Re: Kafka-Streams-Scala for Scala 3

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Matthias,

Yes, it's just a matter of adding the [DISCUSS] prefix in the subject.
By the way, I didn't say this won't need a KIP, just that I won't be
pushing for it, but other maintainers might think it's needed.

For the discuss thread, you should write down what changes in the build and
what steps would be needed to create the artifacts.

Best,

Josep Prat
Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Feb 9, 2024, 19:55 Matthias Berndt <ma...@ttmzero.com>
wrote:

> Hey Josep,
>
> I'm glad you agree that a KIP is not needed here, and I agree with you that
> how to publish these artifacts should be discussed with the Kafka team. In
> fact, this is what I created this thread for 😁 This is my first time
> contributing to Kafka, so I'm going to have to ask what a DISCUSS thread
> is. Is it just a mailing list thread with a subject that starts with
> [DISCUSS], or is there more behind it?
>
> Best regards,
> Matthias
>
> Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat
> <jo...@aiven.io.invalid>:
>
> > Hi Matthias,
> > It's not adding a new functionality but it's changing the way to generate
> > artifacts. In the end we are talking about generating a new binary.
> >
> > I could live with not having a KIP, but a DISCUSS thread I think it's
> > necessary. This signals the community members and maintainers that their
> > input is needed.
> >
> > I could help you with writing the KIP if you want.
> >
> > Best,
> >
> > Josep Prat
> > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Fri, Feb 9, 2024, 18:02 Matthias Berndt <ma...@ttmzero.com>
> > wrote:
> >
> > > Hi Matthias, Hi Josep,
> > >
> > > I'm afraid I can't do the KIP thing as the signup process for Apache
> > > Confluence requires sending me a password reset link via E-Mail and
> said
> > > E-Mail doesn't seem to reach me for some reason. I've contacted the
> > Apache
> > > infrastructure team but haven't yet heard back from them.
> > > That said, I'd like to push back on the notion that a KIP is really
> > > necessary for this change. It's certainly not a “major new feature” as
> it
> > > adds zero extra functionality, and it doesn't affect binary
> compatibility
> > > either as all the currently supported Scala versions are still
> supported.
> > > This looks like a routine upgrade to me. Please, let's try to keep the
> > > administrative overhead to the required minimum, shall we?
> > > Thanks btw for merging Github PR #15239 (removal of
> > scala-collection-compat
> > > dependency from the 2.13 artifact). That will already improve life for
> > > Scala 3 users.
> > >
> > > All the best,
> > > Matthias
> > >
> > >
> > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax <
> > > mjsax@apache.org
> > > >:
> > >
> > > > Josep,
> > > >
> > > > thanks for helping with this. I was also thinking if we might need a
> > KIP
> > > > for this change. Since you had the same though, I would say, yes,
> let's
> > > > do a KIP.
> > > >
> > > > @Matthias: can you prepare a KIP? You can read up on the details on
> the
> > > > wiki page:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >
> > > > If you have any questions about the process, please let us know.
> > > >
> > > > Thanks for pushing this forward!
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 2/8/24 8:08 AM, Matthias Berndt wrote:
> > > > > Hey Josep et al,
> > > > >
> > > > > I've created a ticket regarding this.
> > > > > https://issues.apache.org/jira/browse/KAFKA-16237
> > > > >
> > > > > All the best,
> > > > > Matthias
> > > > >
> > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
> > > > > <jo...@aiven.io.invalid>:
> > > > >>
> > > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us
> > know
> > > > when
> > > > >> your accounts are created and we'll properly set them up so you
> can
> > > > create
> > > > >> and assign tickets to you.
> > > > >>
> > > > >> Best,
> > > > >>
> > > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <
> > > > matthias.berndt@ttmzero.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Thanks Josep, I've applied for a JIRA account and addressed your
> > > > >>> review comments.
> > > > >>>
> > > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> > > > >>> <jo...@aiven.io.invalid>:
> > > > >>>>
> > > > >>>> Hi Matthias,
> > > > >>>>
> > > > >>>> I think for this particular case it would be worth creating a
> JIRA
> > > > ticket
> > > > >>>> for this as it's a new "feature".
> > > > >>>> Regarding the change itself, I think we need to clarify how the
> > > > release
> > > > >>>> process would work. Right now, the script `gradlewAll` is used
> > > (which
> > > > >>>> basically runs the build with Scala version 2.12 and 2.13). If I
> > > > >>> understand
> > > > >>>> your changes correctly, we would need to run the build 3 times:
> > > > >>>> - 1 with property scalaVersion 2.12
> > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> > > > >>>>
> > > > >>>> I think we should document this and discuss when to have this
> > > feature.
> > > > >>> If I
> > > > >>>> remember correctly from when I tried to update Kafka to Scala 3,
> > the
> > > > idea
> > > > >>>> was to push this to a Kafka 4.0 version because we didn't want
> to
> > > > >>> maintain
> > > > >>>> more than 2 Scala versions at the same time. I would encourage
> if
> > > not
> > > > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify
> some
> > of
> > > > >>> these
> > > > >>>> points.
> > > > >>>>
> > > > >>>> I'll add some feedback on the PR itself regarding the changes.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>>
> > > > >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> > > > >>> matthias.berndt@ttmzero.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hi Matthias J., Hi Lucas, Hi Josep,
> > > > >>>>>
> > > > >>>>> Thank you for your encouraging responses regarding a Scala 3
> port
> > > of
> > > > >>>>> Kafka-Streams-Scala, and apologies for the late response from
> my
> > > > side.
> > > > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3
> > > (while
> > > > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the
> > code
> > > > >>>>> were required and the tests also pass. Please take a look and
> let
> > > me
> > > > >>>>> know what you think :-)
> > > > >>>>> https://github.com/apache/kafka/pull/15338
> > > > >>>>>
> > > > >>>>> All the best
> > > > >>>>> Matthias
> > > > >>>>>
> > > > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > > > >>>>> <jo...@aiven.io.invalid>:
> > > > >>>>>>
> > > > >>>>>> Hi,
> > > > >>>>>>
> > > > >>>>>> For reference, prior work on this:
> > > > >>>>>> https://github.com/apache/kafka/pull/11350
> > > > >>>>>> https://github.com/apache/kafka/pull/11432
> > > > >>>>>>
> > > > >>>>>> Best,
> > > > >>>>>>
> > > > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <
> > lbrutschy@confluent.io
> > > > >>>>> .invalid>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Matthiases,
> > > > >>>>>>>
> > > > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes
> > > that
> > > > >>> add
> > > > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be
> > > driven
> > > > >>> by
> > > > >>>>>>> people who use Scala day-to-day, since I believe most Kafka
> > > Streams
> > > > >>>>>>> committers are working with Java.
> > > > >>>>>>>
> > > > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems
> like
> > a
> > > > >>> large
> > > > >>>>>>> undertaking, so option 1 is the first thing we should
> explore.
> > > > >>>>>>>
> > > > >>>>>>> I don't have any experience with Scala 3 migration topics,
> but
> > on
> > > > >>> the
> > > > >>>>>>> Scala website it says
> > > > >>>>>>>> The first piece of good news is that the Scala 3 compiler is
> > > > >>> able to
> > > > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check
> > code
> > > > >>> that
> > > > >>>>>>> depends on modules or libraries compiled with Scala 2.13.
> > > > >>>>>>>> One notable example is the Scala 2.13 library. We have
> indeed
> > > > >>> decided
> > > > >>>>>>> that the Scala 2.13 library is the official standard library
> > for
> > > > >>> Scala
> > > > >>>>> 3.
> > > > >>>>>>> So wouldn't that mean that we are safe in terms of standard
> > > library
> > > > >>>>>>> upgrades if we use core_2.13 in the tests?
> > > > >>>>>>>
> > > > >>>>>>> Cheers,
> > > > >>>>>>> Lucas
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <
> > > mjsax@apache.org>
> > > > >>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module
> > seems
> > > > >>> to
> > > > >>>>> be an
> > > > >>>>>>>> important feature for Kafka Streams and I am generally in
> > favor
> > > > >>> of
> > > > >>>>> your
> > > > >>>>>>>> proposal to add Scala 3 support. However, I am personally no
> > > > >>> Scala
> > > > >>>>>>>> person and it sounds like quite some overhead.
> > > > >>>>>>>>
> > > > >>>>>>>> If you are willing to drive and own this initiative happy to
> > > > >>> support
> > > > >>>>> you
> > > > >>>>>>>> to the extend I can.
> > > > >>>>>>>>
> > > > >>>>>>>> About the concrete proposal: my understanding is that :core
> > will
> > > > >>> move
> > > > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but
> > new
> > > > >>>>> modules
> > > > >>>>>>>> are written in Java only). Thus, down the road the
> > compatibility
> > > > >>>>> issue
> > > > >>>>>>>> would go away naturally, but it's unclear when.
> > > > >>>>>>>>
> > > > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> > > > >>> seems we
> > > > >>>>>>>> could add support for Scala 3 now, taking a risk that it
> might
> > > > >>> break
> > > > >>>>> in
> > > > >>>>>>>> the future assume that the migration off Scala from core is
> > not
> > > > >>> fast
> > > > >>>>>>> enough.
> > > > >>>>>>>>
> > > > >>>>>>>> For proposal (2), I don't think that it would be easily
> > possible
> > > > >>> for
> > > > >>>>>>>> unit/integration tests. We could fall back to system tests
> > > > >>> though,
> > > > >>>>> but
> > > > >>>>>>>> they would be much more heavy weight of course.
> > > > >>>>>>>>
> > > > >>>>>>>> Might be good to hear from others. We might actually also
> want
> > > > >>> to do
> > > > >>>>> a
> > > > >>>>>>>> KIP for this?
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> -Matthias
> > > > >>>>>>>>
> > > > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > >>>>>>>>> Hey there,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I'd like to discuss a Scala 3 port of the
> kafka-streams-scala
> > > > >>>>> library.
> > > > >>>>>>>>> Currently, the build system is set up such that
> > > > >>> kafka-streams-scala
> > > > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same
> > Scala
> > > > >>>>>>>>> compiler versions. This is not an optimal situation because
> > it
> > > > >>>>> means
> > > > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen
> > > > >>>>>>>>> independently of kafka itself. I think this should be
> changed
> > > > >>>>>>>>>
> > > > >>>>>>>>> The production codebase of scala-streams-kafka actually
> > > > >>> compiles
> > > > >>>>> just
> > > > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax
> changes.
> > > > >>> The
> > > > >>>>>>>>> problem is with the tests. These use the
> > `EmbeddedKafkaCluster`
> > > > >>>>> class,
> > > > >>>>>>>>> which means that kafka is pulled into the classpath,
> > > > >>> potentially
> > > > >>>>>>>>> leading to binary compatibility issues.
> > > > >>>>>>>>> I can see several approaches to fixing this:
> > > > >>>>>>>>>
> > > > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible
> > > > >>> version
> > > > >>>>> of
> > > > >>>>>>>>> :core if one is available. Currently, this means that
> > > > >>> everything
> > > > >>>>> can
> > > > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and
> > > > >>> kafka-streams-scala_3
> > > > >>>>>>>>> using core_2.13, as these should be compatible), but when a
> > new
> > > > >>>>>>>>> scala-library version is released that is no longer
> > compatible
> > > > >>> with
> > > > >>>>>>>>> 2.13, we won't be able to test that.
> > > > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> > > > >>> instead
> > > > >>>>>>>>> running the test cluster in a separate JVM or perhaps even
> a
> > > > >>>>>>>>> container.
> > > > >>>>>>>>>
> > > > >>>>>>>>> I'd be willing to get my hands dirty working on this, but
> > > > >>> before I
> > > > >>>>>>>>> start I'd like to get some feedback from the Kafka team
> > > > >>> regarding
> > > > >>>>> the
> > > > >>>>>>>>> approaches outlined above.
> > > > >>>>>>>>>
> > > > >>>>>>>>> All the best
> > > > >>>>>>>>> Matthias Berndt
> > > > >>>>>>>
> > > > >>>>>> KJosep Prat
> > > > >>>>>> Open Source Engineering Director, Aivenjosep.prat@aiven.io
>  |
> > > > >>>>>> +491715557497 | aiven.io
> > > > >>>>>> Aiven Deutschland GmbH
> > > > >>>>>> Alexanderufer 3-7, 10117 Berlin
> > > > >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> [image: Aiven] <https://www.aiven.io>
> > > > >>>>
> > > > >>>> *Josep Prat*
> > > > >>>> Open Source Engineering Director, *Aiven*
> > > > >>>> josep.prat@aiven.io   |   +491715557497
> > > > >>>> aiven.io <https://www.aiven.io>   |   <
> > > > >>> https://www.facebook.com/aivencloud>
> > > > >>>>    <https://www.linkedin.com/company/aiven/>   <
> > > > >>> https://twitter.com/aiven_io>
> > > > >>>> *Aiven Deutschland GmbH*
> > > > >>>> Alexanderufer 3-7, 10117 Berlin
> > > > >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> [image: Aiven] <https://www.aiven.io>
> > > > >>
> > > > >> *Josep Prat*
> > > > >> Open Source Engineering Director, *Aiven*
> > > > >> josep.prat@aiven.io   |   +491715557497
> > > > >> aiven.io <https://www.aiven.io>   |   <
> > > > https://www.facebook.com/aivencloud>
> > > > >>    <https://www.linkedin.com/company/aiven/>   <
> > > > https://twitter.com/aiven_io>
> > > > >> *Aiven Deutschland GmbH*
> > > > >> Alexanderufer 3-7, 10117 Berlin
> > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> >
>

Re: Kafka-Streams-Scala for Scala 3

Posted by Matthias Berndt <ma...@ttmzero.com>.

Hey Josep,

I'm glad you agree that a KIP is not needed here, and I agree with you that
how to publish these artifacts should be discussed with the Kafka team. In
fact, this is what I created this thread for 😁 This is my first time
contributing to Kafka, so I'm going to have to ask what a DISCUSS thread
is. Is it just a mailing list thread with a subject that starts with
[DISCUSS], or is there more behind it?

Best regards,
Matthias

Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat
<jo...@aiven.io.invalid>:

> Hi Matthias,
> It's not adding a new functionality but it's changing the way to generate
> artifacts. In the end we are talking about generating a new binary.
>
> I could live with not having a KIP, but a DISCUSS thread I think it's
> necessary. This signals the community members and maintainers that their
> input is needed.
>
> I could help you with writing the KIP if you want.
>
> Best,
>
> Josep Prat
> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Fri, Feb 9, 2024, 18:02 Matthias Berndt <ma...@ttmzero.com>
> wrote:
>
> > Hi Matthias, Hi Josep,
> >
> > I'm afraid I can't do the KIP thing as the signup process for Apache
> > Confluence requires sending me a password reset link via E-Mail and said
> > E-Mail doesn't seem to reach me for some reason. I've contacted the
> Apache
> > infrastructure team but haven't yet heard back from them.
> > That said, I'd like to push back on the notion that a KIP is really
> > necessary for this change. It's certainly not a “major new feature” as it
> > adds zero extra functionality, and it doesn't affect binary compatibility
> > either as all the currently supported Scala versions are still supported.
> > This looks like a routine upgrade to me. Please, let's try to keep the
> > administrative overhead to the required minimum, shall we?
> > Thanks btw for merging Github PR #15239 (removal of
> scala-collection-compat
> > dependency from the 2.13 artifact). That will already improve life for
> > Scala 3 users.
> >
> > All the best,
> > Matthias
> >
> >
> > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax <
> > mjsax@apache.org
> > >:
> >
> > > Josep,
> > >
> > > thanks for helping with this. I was also thinking if we might need a
> KIP
> > > for this change. Since you had the same though, I would say, yes, let's
> > > do a KIP.
> > >
> > > @Matthias: can you prepare a KIP? You can read up on the details on the
> > > wiki page:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > If you have any questions about the process, please let us know.
> > >
> > > Thanks for pushing this forward!
> > >
> > >
> > > -Matthias
> > >
> > > On 2/8/24 8:08 AM, Matthias Berndt wrote:
> > > > Hey Josep et al,
> > > >
> > > > I've created a ticket regarding this.
> > > > https://issues.apache.org/jira/browse/KAFKA-16237
> > > >
> > > > All the best,
> > > > Matthias
> > > >
> > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
> > > > <jo...@aiven.io.invalid>:
> > > >>
> > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us
> know
> > > when
> > > >> your accounts are created and we'll properly set them up so you can
> > > create
> > > >> and assign tickets to you.
> > > >>
> > > >> Best,
> > > >>
> > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <
> > > matthias.berndt@ttmzero.com>
> > > >> wrote:
> > > >>
> > > >>> Thanks Josep, I've applied for a JIRA account and addressed your
> > > >>> review comments.
> > > >>>
> > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> > > >>> <jo...@aiven.io.invalid>:
> > > >>>>
> > > >>>> Hi Matthias,
> > > >>>>
> > > >>>> I think for this particular case it would be worth creating a JIRA
> > > ticket
> > > >>>> for this as it's a new "feature".
> > > >>>> Regarding the change itself, I think we need to clarify how the
> > > release
> > > >>>> process would work. Right now, the script `gradlewAll` is used
> > (which
> > > >>>> basically runs the build with Scala version 2.12 and 2.13). If I
> > > >>> understand
> > > >>>> your changes correctly, we would need to run the build 3 times:
> > > >>>> - 1 with property scalaVersion 2.12
> > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> > > >>>>
> > > >>>> I think we should document this and discuss when to have this
> > feature.
> > > >>> If I
> > > >>>> remember correctly from when I tried to update Kafka to Scala 3,
> the
> > > idea
> > > >>>> was to push this to a Kafka 4.0 version because we didn't want to
> > > >>> maintain
> > > >>>> more than 2 Scala versions at the same time. I would encourage if
> > not
> > > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify some
> of
> > > >>> these
> > > >>>> points.
> > > >>>>
> > > >>>> I'll add some feedback on the PR itself regarding the changes.
> > > >>>>
> > > >>>> Best,
> > > >>>>
> > > >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> > > >>> matthias.berndt@ttmzero.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Matthias J., Hi Lucas, Hi Josep,
> > > >>>>>
> > > >>>>> Thank you for your encouraging responses regarding a Scala 3 port
> > of
> > > >>>>> Kafka-Streams-Scala, and apologies for the late response from my
> > > side.
> > > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3
> > (while
> > > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the
> code
> > > >>>>> were required and the tests also pass. Please take a look and let
> > me
> > > >>>>> know what you think :-)
> > > >>>>> https://github.com/apache/kafka/pull/15338
> > > >>>>>
> > > >>>>> All the best
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > > >>>>> <jo...@aiven.io.invalid>:
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> For reference, prior work on this:
> > > >>>>>> https://github.com/apache/kafka/pull/11350
> > > >>>>>> https://github.com/apache/kafka/pull/11432
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>>
> > > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <
> lbrutschy@confluent.io
> > > >>>>> .invalid>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Matthiases,
> > > >>>>>>>
> > > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes
> > that
> > > >>> add
> > > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be
> > driven
> > > >>> by
> > > >>>>>>> people who use Scala day-to-day, since I believe most Kafka
> > Streams
> > > >>>>>>> committers are working with Java.
> > > >>>>>>>
> > > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like
> a
> > > >>> large
> > > >>>>>>> undertaking, so option 1 is the first thing we should explore.
> > > >>>>>>>
> > > >>>>>>> I don't have any experience with Scala 3 migration topics, but
> on
> > > >>> the
> > > >>>>>>> Scala website it says
> > > >>>>>>>> The first piece of good news is that the Scala 3 compiler is
> > > >>> able to
> > > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check
> code
> > > >>> that
> > > >>>>>>> depends on modules or libraries compiled with Scala 2.13.
> > > >>>>>>>> One notable example is the Scala 2.13 library. We have indeed
> > > >>> decided
> > > >>>>>>> that the Scala 2.13 library is the official standard library
> for
> > > >>> Scala
> > > >>>>> 3.
> > > >>>>>>> So wouldn't that mean that we are safe in terms of standard
> > library
> > > >>>>>>> upgrades if we use core_2.13 in the tests?
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Lucas
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <
> > mjsax@apache.org>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module
> seems
> > > >>> to
> > > >>>>> be an
> > > >>>>>>>> important feature for Kafka Streams and I am generally in
> favor
> > > >>> of
> > > >>>>> your
> > > >>>>>>>> proposal to add Scala 3 support. However, I am personally no
> > > >>> Scala
> > > >>>>>>>> person and it sounds like quite some overhead.
> > > >>>>>>>>
> > > >>>>>>>> If you are willing to drive and own this initiative happy to
> > > >>> support
> > > >>>>> you
> > > >>>>>>>> to the extend I can.
> > > >>>>>>>>
> > > >>>>>>>> About the concrete proposal: my understanding is that :core
> will
> > > >>> move
> > > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but
> new
> > > >>>>> modules
> > > >>>>>>>> are written in Java only). Thus, down the road the
> compatibility
> > > >>>>> issue
> > > >>>>>>>> would go away naturally, but it's unclear when.
> > > >>>>>>>>
> > > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> > > >>> seems we
> > > >>>>>>>> could add support for Scala 3 now, taking a risk that it might
> > > >>> break
> > > >>>>> in
> > > >>>>>>>> the future assume that the migration off Scala from core is
> not
> > > >>> fast
> > > >>>>>>> enough.
> > > >>>>>>>>
> > > >>>>>>>> For proposal (2), I don't think that it would be easily
> possible
> > > >>> for
> > > >>>>>>>> unit/integration tests. We could fall back to system tests
> > > >>> though,
> > > >>>>> but
> > > >>>>>>>> they would be much more heavy weight of course.
> > > >>>>>>>>
> > > >>>>>>>> Might be good to hear from others. We might actually also want
> > > >>> to do
> > > >>>>> a
> > > >>>>>>>> KIP for this?
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -Matthias
> > > >>>>>>>>
> > > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > >>>>>>>>> Hey there,
> > > >>>>>>>>>
> > > >>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala
> > > >>>>> library.
> > > >>>>>>>>> Currently, the build system is set up such that
> > > >>> kafka-streams-scala
> > > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same
> Scala
> > > >>>>>>>>> compiler versions. This is not an optimal situation because
> it
> > > >>>>> means
> > > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen
> > > >>>>>>>>> independently of kafka itself. I think this should be changed
> > > >>>>>>>>>
> > > >>>>>>>>> The production codebase of scala-streams-kafka actually
> > > >>> compiles
> > > >>>>> just
> > > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes.
> > > >>> The
> > > >>>>>>>>> problem is with the tests. These use the
> `EmbeddedKafkaCluster`
> > > >>>>> class,
> > > >>>>>>>>> which means that kafka is pulled into the classpath,
> > > >>> potentially
> > > >>>>>>>>> leading to binary compatibility issues.
> > > >>>>>>>>> I can see several approaches to fixing this:
> > > >>>>>>>>>
> > > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible
> > > >>> version
> > > >>>>> of
> > > >>>>>>>>> :core if one is available. Currently, this means that
> > > >>> everything
> > > >>>>> can
> > > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and
> > > >>> kafka-streams-scala_3
> > > >>>>>>>>> using core_2.13, as these should be compatible), but when a
> new
> > > >>>>>>>>> scala-library version is released that is no longer
> compatible
> > > >>> with
> > > >>>>>>>>> 2.13, we won't be able to test that.
> > > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> > > >>> instead
> > > >>>>>>>>> running the test cluster in a separate JVM or perhaps even a
> > > >>>>>>>>> container.
> > > >>>>>>>>>
> > > >>>>>>>>> I'd be willing to get my hands dirty working on this, but
> > > >>> before I
> > > >>>>>>>>> start I'd like to get some feedback from the Kafka team
> > > >>> regarding
> > > >>>>> the
> > > >>>>>>>>> approaches outlined above.
> > > >>>>>>>>>
> > > >>>>>>>>> All the best
> > > >>>>>>>>> Matthias Berndt
> > > >>>>>>>
> > > >>>>>> KJosep Prat
> > > >>>>>> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > > >>>>>> +491715557497 | aiven.io
> > > >>>>>> Aiven Deutschland GmbH
> > > >>>>>> Alexanderufer 3-7, 10117 Berlin
> > > >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> [image: Aiven] <https://www.aiven.io>
> > > >>>>
> > > >>>> *Josep Prat*
> > > >>>> Open Source Engineering Director, *Aiven*
> > > >>>> josep.prat@aiven.io   |   +491715557497
> > > >>>> aiven.io <https://www.aiven.io>   |   <
> > > >>> https://www.facebook.com/aivencloud>
> > > >>>>    <https://www.linkedin.com/company/aiven/>   <
> > > >>> https://twitter.com/aiven_io>
> > > >>>> *Aiven Deutschland GmbH*
> > > >>>> Alexanderufer 3-7, 10117 Berlin
> > > >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>>> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> [image: Aiven] <https://www.aiven.io>
> > > >>
> > > >> *Josep Prat*
> > > >> Open Source Engineering Director, *Aiven*
> > > >> josep.prat@aiven.io   |   +491715557497
> > > >> aiven.io <https://www.aiven.io>   |   <
> > > https://www.facebook.com/aivencloud>
> > > >>    <https://www.linkedin.com/company/aiven/>   <
> > > https://twitter.com/aiven_io>
> > > >> *Aiven Deutschland GmbH*
> > > >> Alexanderufer 3-7, 10117 Berlin
> > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: Kafka-Streams-Scala for Scala 3

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Matthias,
It's not adding a new functionality but it's changing the way to generate
artifacts. In the end we are talking about generating a new binary.

I could live with not having a KIP, but a DISCUSS thread I think it's
necessary. This signals the community members and maintainers that their
input is needed.

I could help you with writing the KIP if you want.

Best,

Josep Prat
Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Feb 9, 2024, 18:02 Matthias Berndt <ma...@ttmzero.com>
wrote:

> Hi Matthias, Hi Josep,
>
> I'm afraid I can't do the KIP thing as the signup process for Apache
> Confluence requires sending me a password reset link via E-Mail and said
> E-Mail doesn't seem to reach me for some reason. I've contacted the Apache
> infrastructure team but haven't yet heard back from them.
> That said, I'd like to push back on the notion that a KIP is really
> necessary for this change. It's certainly not a “major new feature” as it
> adds zero extra functionality, and it doesn't affect binary compatibility
> either as all the currently supported Scala versions are still supported.
> This looks like a routine upgrade to me. Please, let's try to keep the
> administrative overhead to the required minimum, shall we?
> Thanks btw for merging Github PR #15239 (removal of scala-collection-compat
> dependency from the 2.13 artifact). That will already improve life for
> Scala 3 users.
>
> All the best,
> Matthias
>
>
> Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax <
> mjsax@apache.org
> >:
>
> > Josep,
> >
> > thanks for helping with this. I was also thinking if we might need a KIP
> > for this change. Since you had the same though, I would say, yes, let's
> > do a KIP.
> >
> > @Matthias: can you prepare a KIP? You can read up on the details on the
> > wiki page:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > If you have any questions about the process, please let us know.
> >
> > Thanks for pushing this forward!
> >
> >
> > -Matthias
> >
> > On 2/8/24 8:08 AM, Matthias Berndt wrote:
> > > Hey Josep et al,
> > >
> > > I've created a ticket regarding this.
> > > https://issues.apache.org/jira/browse/KAFKA-16237
> > >
> > > All the best,
> > > Matthias
> > >
> > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
> > > <jo...@aiven.io.invalid>:
> > >>
> > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know
> > when
> > >> your accounts are created and we'll properly set them up so you can
> > create
> > >> and assign tickets to you.
> > >>
> > >> Best,
> > >>
> > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <
> > matthias.berndt@ttmzero.com>
> > >> wrote:
> > >>
> > >>> Thanks Josep, I've applied for a JIRA account and addressed your
> > >>> review comments.
> > >>>
> > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> > >>> <jo...@aiven.io.invalid>:
> > >>>>
> > >>>> Hi Matthias,
> > >>>>
> > >>>> I think for this particular case it would be worth creating a JIRA
> > ticket
> > >>>> for this as it's a new "feature".
> > >>>> Regarding the change itself, I think we need to clarify how the
> > release
> > >>>> process would work. Right now, the script `gradlewAll` is used
> (which
> > >>>> basically runs the build with Scala version 2.12 and 2.13). If I
> > >>> understand
> > >>>> your changes correctly, we would need to run the build 3 times:
> > >>>> - 1 with property scalaVersion 2.12
> > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> > >>>>
> > >>>> I think we should document this and discuss when to have this
> feature.
> > >>> If I
> > >>>> remember correctly from when I tried to update Kafka to Scala 3, the
> > idea
> > >>>> was to push this to a Kafka 4.0 version because we didn't want to
> > >>> maintain
> > >>>> more than 2 Scala versions at the same time. I would encourage if
> not
> > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify some of
> > >>> these
> > >>>> points.
> > >>>>
> > >>>> I'll add some feedback on the PR itself regarding the changes.
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> > >>> matthias.berndt@ttmzero.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Matthias J., Hi Lucas, Hi Josep,
> > >>>>>
> > >>>>> Thank you for your encouraging responses regarding a Scala 3 port
> of
> > >>>>> Kafka-Streams-Scala, and apologies for the late response from my
> > side.
> > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3
> (while
> > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the code
> > >>>>> were required and the tests also pass. Please take a look and let
> me
> > >>>>> know what you think :-)
> > >>>>> https://github.com/apache/kafka/pull/15338
> > >>>>>
> > >>>>> All the best
> > >>>>> Matthias
> > >>>>>
> > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > >>>>> <jo...@aiven.io.invalid>:
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> For reference, prior work on this:
> > >>>>>> https://github.com/apache/kafka/pull/11350
> > >>>>>> https://github.com/apache/kafka/pull/11432
> > >>>>>>
> > >>>>>> Best,
> > >>>>>>
> > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> > >>>>> .invalid>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Matthiases,
> > >>>>>>>
> > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes
> that
> > >>> add
> > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be
> driven
> > >>> by
> > >>>>>>> people who use Scala day-to-day, since I believe most Kafka
> Streams
> > >>>>>>> committers are working with Java.
> > >>>>>>>
> > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like a
> > >>> large
> > >>>>>>> undertaking, so option 1 is the first thing we should explore.
> > >>>>>>>
> > >>>>>>> I don't have any experience with Scala 3 migration topics, but on
> > >>> the
> > >>>>>>> Scala website it says
> > >>>>>>>> The first piece of good news is that the Scala 3 compiler is
> > >>> able to
> > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check code
> > >>> that
> > >>>>>>> depends on modules or libraries compiled with Scala 2.13.
> > >>>>>>>> One notable example is the Scala 2.13 library. We have indeed
> > >>> decided
> > >>>>>>> that the Scala 2.13 library is the official standard library for
> > >>> Scala
> > >>>>> 3.
> > >>>>>>> So wouldn't that mean that we are safe in terms of standard
> library
> > >>>>>>> upgrades if we use core_2.13 in the tests?
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Lucas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <
> mjsax@apache.org>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module seems
> > >>> to
> > >>>>> be an
> > >>>>>>>> important feature for Kafka Streams and I am generally in favor
> > >>> of
> > >>>>> your
> > >>>>>>>> proposal to add Scala 3 support. However, I am personally no
> > >>> Scala
> > >>>>>>>> person and it sounds like quite some overhead.
> > >>>>>>>>
> > >>>>>>>> If you are willing to drive and own this initiative happy to
> > >>> support
> > >>>>> you
> > >>>>>>>> to the extend I can.
> > >>>>>>>>
> > >>>>>>>> About the concrete proposal: my understanding is that :core will
> > >>> move
> > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but new
> > >>>>> modules
> > >>>>>>>> are written in Java only). Thus, down the road the compatibility
> > >>>>> issue
> > >>>>>>>> would go away naturally, but it's unclear when.
> > >>>>>>>>
> > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> > >>> seems we
> > >>>>>>>> could add support for Scala 3 now, taking a risk that it might
> > >>> break
> > >>>>> in
> > >>>>>>>> the future assume that the migration off Scala from core is not
> > >>> fast
> > >>>>>>> enough.
> > >>>>>>>>
> > >>>>>>>> For proposal (2), I don't think that it would be easily possible
> > >>> for
> > >>>>>>>> unit/integration tests. We could fall back to system tests
> > >>> though,
> > >>>>> but
> > >>>>>>>> they would be much more heavy weight of course.
> > >>>>>>>>
> > >>>>>>>> Might be good to hear from others. We might actually also want
> > >>> to do
> > >>>>> a
> > >>>>>>>> KIP for this?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -Matthias
> > >>>>>>>>
> > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > >>>>>>>>> Hey there,
> > >>>>>>>>>
> > >>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala
> > >>>>> library.
> > >>>>>>>>> Currently, the build system is set up such that
> > >>> kafka-streams-scala
> > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same Scala
> > >>>>>>>>> compiler versions. This is not an optimal situation because it
> > >>>>> means
> > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen
> > >>>>>>>>> independently of kafka itself. I think this should be changed
> > >>>>>>>>>
> > >>>>>>>>> The production codebase of scala-streams-kafka actually
> > >>> compiles
> > >>>>> just
> > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes.
> > >>> The
> > >>>>>>>>> problem is with the tests. These use the `EmbeddedKafkaCluster`
> > >>>>> class,
> > >>>>>>>>> which means that kafka is pulled into the classpath,
> > >>> potentially
> > >>>>>>>>> leading to binary compatibility issues.
> > >>>>>>>>> I can see several approaches to fixing this:
> > >>>>>>>>>
> > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible
> > >>> version
> > >>>>> of
> > >>>>>>>>> :core if one is available. Currently, this means that
> > >>> everything
> > >>>>> can
> > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12,
> > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and
> > >>> kafka-streams-scala_3
> > >>>>>>>>> using core_2.13, as these should be compatible), but when a new
> > >>>>>>>>> scala-library version is released that is no longer compatible
> > >>> with
> > >>>>>>>>> 2.13, we won't be able to test that.
> > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> > >>> instead
> > >>>>>>>>> running the test cluster in a separate JVM or perhaps even a
> > >>>>>>>>> container.
> > >>>>>>>>>
> > >>>>>>>>> I'd be willing to get my hands dirty working on this, but
> > >>> before I
> > >>>>>>>>> start I'd like to get some feedback from the Kafka team
> > >>> regarding
> > >>>>> the
> > >>>>>>>>> approaches outlined above.
> > >>>>>>>>>
> > >>>>>>>>> All the best
> > >>>>>>>>> Matthias Berndt
> > >>>>>>>
> > >>>>>> KJosep Prat
> > >>>>>> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > >>>>>> +491715557497 | aiven.io
> > >>>>>> Aiven Deutschland GmbH
> > >>>>>> Alexanderufer 3-7, 10117 Berlin
> > >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> [image: Aiven] <https://www.aiven.io>
> > >>>>
> > >>>> *Josep Prat*
> > >>>> Open Source Engineering Director, *Aiven*
> > >>>> josep.prat@aiven.io   |   +491715557497
> > >>>> aiven.io <https://www.aiven.io>   |   <
> > >>> https://www.facebook.com/aivencloud>
> > >>>>    <https://www.linkedin.com/company/aiven/>   <
> > >>> https://twitter.com/aiven_io>
> > >>>> *Aiven Deutschland GmbH*
> > >>>> Alexanderufer 3-7, 10117 Berlin
> > >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> > >>
> > >>
> > >> --
> > >> [image: Aiven] <https://www.aiven.io>
> > >>
> > >> *Josep Prat*
> > >> Open Source Engineering Director, *Aiven*
> > >> josep.prat@aiven.io   |   +491715557497
> > >> aiven.io <https://www.aiven.io>   |   <
> > https://www.facebook.com/aivencloud>
> > >>    <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > >> *Aiven Deutschland GmbH*
> > >> Alexanderufer 3-7, 10117 Berlin
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> >
>

Re: Kafka-Streams-Scala for Scala 3

Posted by Matthias Berndt <ma...@ttmzero.com>.

Hi Matthias, Hi Josep,

I'm afraid I can't do the KIP thing as the signup process for Apache
Confluence requires sending me a password reset link via E-Mail and said
E-Mail doesn't seem to reach me for some reason. I've contacted the Apache
infrastructure team but haven't yet heard back from them.
That said, I'd like to push back on the notion that a KIP is really
necessary for this change. It's certainly not a “major new feature” as it
adds zero extra functionality, and it doesn't affect binary compatibility
either as all the currently supported Scala versions are still supported.
This looks like a routine upgrade to me. Please, let's try to keep the
administrative overhead to the required minimum, shall we?
Thanks btw for merging Github PR #15239 (removal of scala-collection-compat
dependency from the 2.13 artifact). That will already improve life for
Scala 3 users.

All the best,
Matthias


Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax <mjsax@apache.org
>:

> Josep,
>
> thanks for helping with this. I was also thinking if we might need a KIP
> for this change. Since you had the same though, I would say, yes, let's
> do a KIP.
>
> @Matthias: can you prepare a KIP? You can read up on the details on the
> wiki page:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> If you have any questions about the process, please let us know.
>
> Thanks for pushing this forward!
>
>
> -Matthias
>
> On 2/8/24 8:08 AM, Matthias Berndt wrote:
> > Hey Josep et al,
> >
> > I've created a ticket regarding this.
> > https://issues.apache.org/jira/browse/KAFKA-16237
> >
> > All the best,
> > Matthias
> >
> > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
> > <jo...@aiven.io.invalid>:
> >>
> >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know
> when
> >> your accounts are created and we'll properly set them up so you can
> create
> >> and assign tickets to you.
> >>
> >> Best,
> >>
> >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <
> matthias.berndt@ttmzero.com>
> >> wrote:
> >>
> >>> Thanks Josep, I've applied for a JIRA account and addressed your
> >>> review comments.
> >>>
> >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> >>> <jo...@aiven.io.invalid>:
> >>>>
> >>>> Hi Matthias,
> >>>>
> >>>> I think for this particular case it would be worth creating a JIRA
> ticket
> >>>> for this as it's a new "feature".
> >>>> Regarding the change itself, I think we need to clarify how the
> release
> >>>> process would work. Right now, the script `gradlewAll` is used (which
> >>>> basically runs the build with Scala version 2.12 and 2.13). If I
> >>> understand
> >>>> your changes correctly, we would need to run the build 3 times:
> >>>> - 1 with property scalaVersion 2.12
> >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> >>>>
> >>>> I think we should document this and discuss when to have this feature.
> >>> If I
> >>>> remember correctly from when I tried to update Kafka to Scala 3, the
> idea
> >>>> was to push this to a Kafka 4.0 version because we didn't want to
> >>> maintain
> >>>> more than 2 Scala versions at the same time. I would encourage if not
> >>>> having a KIP, at least open up a [DISCUSS] thread to clarify some of
> >>> these
> >>>> points.
> >>>>
> >>>> I'll add some feedback on the PR itself regarding the changes.
> >>>>
> >>>> Best,
> >>>>
> >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> >>> matthias.berndt@ttmzero.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Matthias J., Hi Lucas, Hi Josep,
> >>>>>
> >>>>> Thank you for your encouraging responses regarding a Scala 3 port of
> >>>>> Kafka-Streams-Scala, and apologies for the late response from my
> side.
> >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
> >>>>> retaining support for 2.13 and 2.12). Almost no changes to the code
> >>>>> were required and the tests also pass. Please take a look and let me
> >>>>> know what you think :-)
> >>>>> https://github.com/apache/kafka/pull/15338
> >>>>>
> >>>>> All the best
> >>>>> Matthias
> >>>>>
> >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> >>>>> <jo...@aiven.io.invalid>:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> For reference, prior work on this:
> >>>>>> https://github.com/apache/kafka/pull/11350
> >>>>>> https://github.com/apache/kafka/pull/11432
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Matthiases,
> >>>>>>>
> >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes that
> >>> add
> >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be driven
> >>> by
> >>>>>>> people who use Scala day-to-day, since I believe most Kafka Streams
> >>>>>>> committers are working with Java.
> >>>>>>>
> >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like a
> >>> large
> >>>>>>> undertaking, so option 1 is the first thing we should explore.
> >>>>>>>
> >>>>>>> I don't have any experience with Scala 3 migration topics, but on
> >>> the
> >>>>>>> Scala website it says
> >>>>>>>> The first piece of good news is that the Scala 3 compiler is
> >>> able to
> >>>>>>> read the Scala 2.13 Pickle format and thus it can type check code
> >>> that
> >>>>>>> depends on modules or libraries compiled with Scala 2.13.
> >>>>>>>> One notable example is the Scala 2.13 library. We have indeed
> >>> decided
> >>>>>>> that the Scala 2.13 library is the official standard library for
> >>> Scala
> >>>>> 3.
> >>>>>>> So wouldn't that mean that we are safe in terms of standard library
> >>>>>>> upgrades if we use core_2.13 in the tests?
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Lucas
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module seems
> >>> to
> >>>>> be an
> >>>>>>>> important feature for Kafka Streams and I am generally in favor
> >>> of
> >>>>> your
> >>>>>>>> proposal to add Scala 3 support. However, I am personally no
> >>> Scala
> >>>>>>>> person and it sounds like quite some overhead.
> >>>>>>>>
> >>>>>>>> If you are willing to drive and own this initiative happy to
> >>> support
> >>>>> you
> >>>>>>>> to the extend I can.
> >>>>>>>>
> >>>>>>>> About the concrete proposal: my understanding is that :core will
> >>> move
> >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but new
> >>>>> modules
> >>>>>>>> are written in Java only). Thus, down the road the compatibility
> >>>>> issue
> >>>>>>>> would go away naturally, but it's unclear when.
> >>>>>>>>
> >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> >>> seems we
> >>>>>>>> could add support for Scala 3 now, taking a risk that it might
> >>> break
> >>>>> in
> >>>>>>>> the future assume that the migration off Scala from core is not
> >>> fast
> >>>>>>> enough.
> >>>>>>>>
> >>>>>>>> For proposal (2), I don't think that it would be easily possible
> >>> for
> >>>>>>>> unit/integration tests. We could fall back to system tests
> >>> though,
> >>>>> but
> >>>>>>>> they would be much more heavy weight of course.
> >>>>>>>>
> >>>>>>>> Might be good to hear from others. We might actually also want
> >>> to do
> >>>>> a
> >>>>>>>> KIP for this?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote:
> >>>>>>>>> Hey there,
> >>>>>>>>>
> >>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala
> >>>>> library.
> >>>>>>>>> Currently, the build system is set up such that
> >>> kafka-streams-scala
> >>>>>>>>> and core (i. e. kafka itself) are compiled with the same Scala
> >>>>>>>>> compiler versions. This is not an optimal situation because it
> >>>>> means
> >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen
> >>>>>>>>> independently of kafka itself. I think this should be changed
> >>>>>>>>>
> >>>>>>>>> The production codebase of scala-streams-kafka actually
> >>> compiles
> >>>>> just
> >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes.
> >>> The
> >>>>>>>>> problem is with the tests. These use the `EmbeddedKafkaCluster`
> >>>>> class,
> >>>>>>>>> which means that kafka is pulled into the classpath,
> >>> potentially
> >>>>>>>>> leading to binary compatibility issues.
> >>>>>>>>> I can see several approaches to fixing this:
> >>>>>>>>>
> >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible
> >>> version
> >>>>> of
> >>>>>>>>> :core if one is available. Currently, this means that
> >>> everything
> >>>>> can
> >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12,
> >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and
> >>> kafka-streams-scala_3
> >>>>>>>>> using core_2.13, as these should be compatible), but when a new
> >>>>>>>>> scala-library version is released that is no longer compatible
> >>> with
> >>>>>>>>> 2.13, we won't be able to test that.
> >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> >>> instead
> >>>>>>>>> running the test cluster in a separate JVM or perhaps even a
> >>>>>>>>> container.
> >>>>>>>>>
> >>>>>>>>> I'd be willing to get my hands dirty working on this, but
> >>> before I
> >>>>>>>>> start I'd like to get some feedback from the Kafka team
> >>> regarding
> >>>>> the
> >>>>>>>>> approaches outlined above.
> >>>>>>>>>
> >>>>>>>>> All the best
> >>>>>>>>> Matthias Berndt
> >>>>>>>
> >>>>>> KJosep Prat
> >>>>>> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> >>>>>> +491715557497 | aiven.io
> >>>>>> Aiven Deutschland GmbH
> >>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> [image: Aiven] <https://www.aiven.io>
> >>>>
> >>>> *Josep Prat*
> >>>> Open Source Engineering Director, *Aiven*
> >>>> josep.prat@aiven.io   |   +491715557497
> >>>> aiven.io <https://www.aiven.io>   |   <
> >>> https://www.facebook.com/aivencloud>
> >>>>    <https://www.linkedin.com/company/aiven/>   <
> >>> https://twitter.com/aiven_io>
> >>>> *Aiven Deutschland GmbH*
> >>>> Alexanderufer 3-7, 10117 Berlin
> >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>
> >>
> >> --
> >> [image: Aiven] <https://www.aiven.io>
> >>
> >> *Josep Prat*
> >> Open Source Engineering Director, *Aiven*
> >> josep.prat@aiven.io   |   +491715557497
> >> aiven.io <https://www.aiven.io>   |   <
> https://www.facebook.com/aivencloud>
> >>    <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> >> *Aiven Deutschland GmbH*
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: Kafka-Streams-Scala for Scala 3

Posted by "Matthias J. Sax" <mj...@apache.org>.

Josep,

thanks for helping with this. I was also thinking if we might need a KIP 
for this change. Since you had the same though, I would say, yes, let's 
do a KIP.

@Matthias: can you prepare a KIP? You can read up on the details on the 
wiki page: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

If you have any questions about the process, please let us know.

Thanks for pushing this forward!


-Matthias

On 2/8/24 8:08 AM, Matthias Berndt wrote:
> Hey Josep et al,
> 
> I've created a ticket regarding this.
> https://issues.apache.org/jira/browse/KAFKA-16237
> 
> All the best,
> Matthias
> 
> Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
> <jo...@aiven.io.invalid>:
>>
>> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
>> your accounts are created and we'll properly set them up so you can create
>> and assign tickets to you.
>>
>> Best,
>>
>> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <ma...@ttmzero.com>
>> wrote:
>>
>>> Thanks Josep, I've applied for a JIRA account and addressed your
>>> review comments.
>>>
>>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
>>> <jo...@aiven.io.invalid>:
>>>>
>>>> Hi Matthias,
>>>>
>>>> I think for this particular case it would be worth creating a JIRA ticket
>>>> for this as it's a new "feature".
>>>> Regarding the change itself, I think we need to clarify how the release
>>>> process would work. Right now, the script `gradlewAll` is used (which
>>>> basically runs the build with Scala version 2.12 and 2.13). If I
>>> understand
>>>> your changes correctly, we would need to run the build 3 times:
>>>> - 1 with property scalaVersion 2.12
>>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
>>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
>>>>
>>>> I think we should document this and discuss when to have this feature.
>>> If I
>>>> remember correctly from when I tried to update Kafka to Scala 3, the idea
>>>> was to push this to a Kafka 4.0 version because we didn't want to
>>> maintain
>>>> more than 2 Scala versions at the same time. I would encourage if not
>>>> having a KIP, at least open up a [DISCUSS] thread to clarify some of
>>> these
>>>> points.
>>>>
>>>> I'll add some feedback on the PR itself regarding the changes.
>>>>
>>>> Best,
>>>>
>>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
>>> matthias.berndt@ttmzero.com>
>>>> wrote:
>>>>
>>>>> Hi Matthias J., Hi Lucas, Hi Josep,
>>>>>
>>>>> Thank you for your encouraging responses regarding a Scala 3 port of
>>>>> Kafka-Streams-Scala, and apologies for the late response from my side.
>>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
>>>>> retaining support for 2.13 and 2.12). Almost no changes to the code
>>>>> were required and the tests also pass. Please take a look and let me
>>>>> know what you think :-)
>>>>> https://github.com/apache/kafka/pull/15338
>>>>>
>>>>> All the best
>>>>> Matthias
>>>>>
>>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
>>>>> <jo...@aiven.io.invalid>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> For reference, prior work on this:
>>>>>> https://github.com/apache/kafka/pull/11350
>>>>>> https://github.com/apache/kafka/pull/11432
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
>>>>> .invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Matthiases,
>>>>>>>
>>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes that
>>> add
>>>>>>> Scala 3 support. However, as Matthias S. said, it has to be driven
>>> by
>>>>>>> people who use Scala day-to-day, since I believe most Kafka Streams
>>>>>>> committers are working with Java.
>>>>>>>
>>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like a
>>> large
>>>>>>> undertaking, so option 1 is the first thing we should explore.
>>>>>>>
>>>>>>> I don't have any experience with Scala 3 migration topics, but on
>>> the
>>>>>>> Scala website it says
>>>>>>>> The first piece of good news is that the Scala 3 compiler is
>>> able to
>>>>>>> read the Scala 2.13 Pickle format and thus it can type check code
>>> that
>>>>>>> depends on modules or libraries compiled with Scala 2.13.
>>>>>>>> One notable example is the Scala 2.13 library. We have indeed
>>> decided
>>>>>>> that the Scala 2.13 library is the official standard library for
>>> Scala
>>>>> 3.
>>>>>>> So wouldn't that mean that we are safe in terms of standard library
>>>>>>> upgrades if we use core_2.13 in the tests?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Lucas
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
>>>>> wrote:
>>>>>>>>
>>>>>>>> Thanks for raising this. The `kafka-streams-scala` module seems
>>> to
>>>>> be an
>>>>>>>> important feature for Kafka Streams and I am generally in favor
>>> of
>>>>> your
>>>>>>>> proposal to add Scala 3 support. However, I am personally no
>>> Scala
>>>>>>>> person and it sounds like quite some overhead.
>>>>>>>>
>>>>>>>> If you are willing to drive and own this initiative happy to
>>> support
>>>>> you
>>>>>>>> to the extend I can.
>>>>>>>>
>>>>>>>> About the concrete proposal: my understanding is that :core will
>>> move
>>>>>>>> off Scala long-term (not 100% sure what the timeline is, but new
>>>>> modules
>>>>>>>> are written in Java only). Thus, down the road the compatibility
>>>>> issue
>>>>>>>> would go away naturally, but it's unclear when.
>>>>>>>>
>>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it
>>> seems we
>>>>>>>> could add support for Scala 3 now, taking a risk that it might
>>> break
>>>>> in
>>>>>>>> the future assume that the migration off Scala from core is not
>>> fast
>>>>>>> enough.
>>>>>>>>
>>>>>>>> For proposal (2), I don't think that it would be easily possible
>>> for
>>>>>>>> unit/integration tests. We could fall back to system tests
>>> though,
>>>>> but
>>>>>>>> they would be much more heavy weight of course.
>>>>>>>>
>>>>>>>> Might be good to hear from others. We might actually also want
>>> to do
>>>>> a
>>>>>>>> KIP for this?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote:
>>>>>>>>> Hey there,
>>>>>>>>>
>>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala
>>>>> library.
>>>>>>>>> Currently, the build system is set up such that
>>> kafka-streams-scala
>>>>>>>>> and core (i. e. kafka itself) are compiled with the same Scala
>>>>>>>>> compiler versions. This is not an optimal situation because it
>>>>> means
>>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen
>>>>>>>>> independently of kafka itself. I think this should be changed
>>>>>>>>>
>>>>>>>>> The production codebase of scala-streams-kafka actually
>>> compiles
>>>>> just
>>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes.
>>> The
>>>>>>>>> problem is with the tests. These use the `EmbeddedKafkaCluster`
>>>>> class,
>>>>>>>>> which means that kafka is pulled into the classpath,
>>> potentially
>>>>>>>>> leading to binary compatibility issues.
>>>>>>>>> I can see several approaches to fixing this:
>>>>>>>>>
>>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible
>>> version
>>>>> of
>>>>>>>>> :core if one is available. Currently, this means that
>>> everything
>>>>> can
>>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12,
>>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and
>>> kafka-streams-scala_3
>>>>>>>>> using core_2.13, as these should be compatible), but when a new
>>>>>>>>> scala-library version is released that is no longer compatible
>>> with
>>>>>>>>> 2.13, we won't be able to test that.
>>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster,
>>> instead
>>>>>>>>> running the test cluster in a separate JVM or perhaps even a
>>>>>>>>> container.
>>>>>>>>>
>>>>>>>>> I'd be willing to get my hands dirty working on this, but
>>> before I
>>>>>>>>> start I'd like to get some feedback from the Kafka team
>>> regarding
>>>>> the
>>>>>>>>> approaches outlined above.
>>>>>>>>>
>>>>>>>>> All the best
>>>>>>>>> Matthias Berndt
>>>>>>>
>>>>>> KJosep Prat
>>>>>> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
>>>>>> +491715557497 | aiven.io
>>>>>> Aiven Deutschland GmbH
>>>>>> Alexanderufer 3-7, 10117 Berlin
>>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>>>
>>>>
>>>>
>>>> --
>>>> [image: Aiven] <https://www.aiven.io>
>>>>
>>>> *Josep Prat*
>>>> Open Source Engineering Director, *Aiven*
>>>> josep.prat@aiven.io   |   +491715557497
>>>> aiven.io <https://www.aiven.io>   |   <
>>> https://www.facebook.com/aivencloud>
>>>>    <https://www.linkedin.com/company/aiven/>   <
>>> https://twitter.com/aiven_io>
>>>> *Aiven Deutschland GmbH*
>>>> Alexanderufer 3-7, 10117 Berlin
>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>
>>
>>
>> --
>> [image: Aiven] <https://www.aiven.io>
>>
>> *Josep Prat*
>> Open Source Engineering Director, *Aiven*
>> josep.prat@aiven.io   |   +491715557497
>> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
>>    <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Matthias Berndt <ma...@ttmzero.com>.

Hey Josep et al,

I've created a ticket regarding this.
https://issues.apache.org/jira/browse/KAFKA-16237

All the best,
Matthias

Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
<jo...@aiven.io.invalid>:
>
> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
> your accounts are created and we'll properly set them up so you can create
> and assign tickets to you.
>
> Best,
>
> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <ma...@ttmzero.com>
> wrote:
>
> > Thanks Josep, I've applied for a JIRA account and addressed your
> > review comments.
> >
> > Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> > <jo...@aiven.io.invalid>:
> > >
> > > Hi Matthias,
> > >
> > > I think for this particular case it would be worth creating a JIRA ticket
> > > for this as it's a new "feature".
> > > Regarding the change itself, I think we need to clarify how the release
> > > process would work. Right now, the script `gradlewAll` is used (which
> > > basically runs the build with Scala version 2.12 and 2.13). If I
> > understand
> > > your changes correctly, we would need to run the build 3 times:
> > > - 1 with property scalaVersion 2.12
> > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> > >
> > > I think we should document this and discuss when to have this feature.
> > If I
> > > remember correctly from when I tried to update Kafka to Scala 3, the idea
> > > was to push this to a Kafka 4.0 version because we didn't want to
> > maintain
> > > more than 2 Scala versions at the same time. I would encourage if not
> > > having a KIP, at least open up a [DISCUSS] thread to clarify some of
> > these
> > > points.
> > >
> > > I'll add some feedback on the PR itself regarding the changes.
> > >
> > > Best,
> > >
> > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> > matthias.berndt@ttmzero.com>
> > > wrote:
> > >
> > > > Hi Matthias J., Hi Lucas, Hi Josep,
> > > >
> > > > Thank you for your encouraging responses regarding a Scala 3 port of
> > > > Kafka-Streams-Scala, and apologies for the late response from my side.
> > > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
> > > > retaining support for 2.13 and 2.12). Almost no changes to the code
> > > > were required and the tests also pass. Please take a look and let me
> > > > know what you think :-)
> > > > https://github.com/apache/kafka/pull/15338
> > > >
> > > > All the best
> > > > Matthias
> > > >
> > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > > > <jo...@aiven.io.invalid>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > For reference, prior work on this:
> > > > > https://github.com/apache/kafka/pull/11350
> > > > > https://github.com/apache/kafka/pull/11432
> > > > >
> > > > > Best,
> > > > >
> > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi Matthiases,
> > > > > >
> > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that
> > add
> > > > > > Scala 3 support. However, as Matthias S. said, it has to be driven
> > by
> > > > > > people who use Scala day-to-day, since I believe most Kafka Streams
> > > > > > committers are working with Java.
> > > > > >
> > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a
> > large
> > > > > > undertaking, so option 1 is the first thing we should explore.
> > > > > >
> > > > > > I don't have any experience with Scala 3 migration topics, but on
> > the
> > > > > > Scala website it says
> > > > > > > The first piece of good news is that the Scala 3 compiler is
> > able to
> > > > > > read the Scala 2.13 Pickle format and thus it can type check code
> > that
> > > > > > depends on modules or libraries compiled with Scala 2.13.
> > > > > > > One notable example is the Scala 2.13 library. We have indeed
> > decided
> > > > > > that the Scala 2.13 library is the official standard library for
> > Scala
> > > > 3.
> > > > > > So wouldn't that mean that we are safe in terms of standard library
> > > > > > upgrades if we use core_2.13 in the tests?
> > > > > >
> > > > > > Cheers,
> > > > > > Lucas
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems
> > to
> > > > be an
> > > > > > > important feature for Kafka Streams and I am generally in favor
> > of
> > > > your
> > > > > > > proposal to add Scala 3 support. However, I am personally no
> > Scala
> > > > > > > person and it sounds like quite some overhead.
> > > > > > >
> > > > > > > If you are willing to drive and own this initiative happy to
> > support
> > > > you
> > > > > > > to the extend I can.
> > > > > > >
> > > > > > > About the concrete proposal: my understanding is that :core will
> > move
> > > > > > > off Scala long-term (not 100% sure what the timeline is, but new
> > > > modules
> > > > > > > are written in Java only). Thus, down the road the compatibility
> > > > issue
> > > > > > > would go away naturally, but it's unclear when.
> > > > > > >
> > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> > seems we
> > > > > > > could add support for Scala 3 now, taking a risk that it might
> > break
> > > > in
> > > > > > > the future assume that the migration off Scala from core is not
> > fast
> > > > > > enough.
> > > > > > >
> > > > > > > For proposal (2), I don't think that it would be easily possible
> > for
> > > > > > > unit/integration tests. We could fall back to system tests
> > though,
> > > > but
> > > > > > > they would be much more heavy weight of course.
> > > > > > >
> > > > > > > Might be good to hear from others. We might actually also want
> > to do
> > > > a
> > > > > > > KIP for this?
> > > > > > >
> > > > > > >
> > > > > > > -Matthias
> > > > > > >
> > > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > > > > > Hey there,
> > > > > > > >
> > > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala
> > > > library.
> > > > > > > > Currently, the build system is set up such that
> > kafka-streams-scala
> > > > > > > > and core (i. e. kafka itself) are compiled with the same Scala
> > > > > > > > compiler versions. This is not an optimal situation because it
> > > > means
> > > > > > > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > > > > > > independently of kafka itself. I think this should be changed
> > > > > > > >
> > > > > > > > The production codebase of scala-streams-kafka actually
> > compiles
> > > > just
> > > > > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes.
> > The
> > > > > > > > problem is with the tests. These use the `EmbeddedKafkaCluster`
> > > > class,
> > > > > > > > which means that kafka is pulled into the classpath,
> > potentially
> > > > > > > > leading to binary compatibility issues.
> > > > > > > > I can see several approaches to fixing this:
> > > > > > > >
> > > > > > > > 1. Run the kafka-streams-scala tests using the compatible
> > version
> > > > of
> > > > > > > > :core if one is available. Currently, this means that
> > everything
> > > > can
> > > > > > > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > > > > > kafka-streams-scala_2.13 using core_2.13 and
> > kafka-streams-scala_3
> > > > > > > > using core_2.13, as these should be compatible), but when a new
> > > > > > > > scala-library version is released that is no longer compatible
> > with
> > > > > > > > 2.13, we won't be able to test that.
> > > > > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> > instead
> > > > > > > > running the test cluster in a separate JVM or perhaps even a
> > > > > > > > container.
> > > > > > > >
> > > > > > > > I'd be willing to get my hands dirty working on this, but
> > before I
> > > > > > > > start I'd like to get some feedback from the Kafka team
> > regarding
> > > > the
> > > > > > > > approaches outlined above.
> > > > > > > >
> > > > > > > > All the best
> > > > > > > > Matthias Berndt
> > > > > >
> > > > > KJosep Prat
> > > > > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > > > > +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] <https://www.aiven.io>
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.prat@aiven.io   |   +491715557497
> > > aiven.io <https://www.aiven.io>   |   <
> > https://www.facebook.com/aivencloud>
> > >   <https://www.linkedin.com/company/aiven/>   <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
>   <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
your accounts are created and we'll properly set them up so you can create
and assign tickets to you.

Best,

On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <ma...@ttmzero.com>
wrote:

> Thanks Josep, I've applied for a JIRA account and addressed your
> review comments.
>
> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
> <jo...@aiven.io.invalid>:
> >
> > Hi Matthias,
> >
> > I think for this particular case it would be worth creating a JIRA ticket
> > for this as it's a new "feature".
> > Regarding the change itself, I think we need to clarify how the release
> > process would work. Right now, the script `gradlewAll` is used (which
> > basically runs the build with Scala version 2.12 and 2.13). If I
> understand
> > your changes correctly, we would need to run the build 3 times:
> > - 1 with property scalaVersion 2.12
> > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
> >
> > I think we should document this and discuss when to have this feature.
> If I
> > remember correctly from when I tried to update Kafka to Scala 3, the idea
> > was to push this to a Kafka 4.0 version because we didn't want to
> maintain
> > more than 2 Scala versions at the same time. I would encourage if not
> > having a KIP, at least open up a [DISCUSS] thread to clarify some of
> these
> > points.
> >
> > I'll add some feedback on the PR itself regarding the changes.
> >
> > Best,
> >
> > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
> matthias.berndt@ttmzero.com>
> > wrote:
> >
> > > Hi Matthias J., Hi Lucas, Hi Josep,
> > >
> > > Thank you for your encouraging responses regarding a Scala 3 port of
> > > Kafka-Streams-Scala, and apologies for the late response from my side.
> > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
> > > retaining support for 2.13 and 2.12). Almost no changes to the code
> > > were required and the tests also pass. Please take a look and let me
> > > know what you think :-)
> > > https://github.com/apache/kafka/pull/15338
> > >
> > > All the best
> > > Matthias
> > >
> > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > > <jo...@aiven.io.invalid>:
> > > >
> > > > Hi,
> > > >
> > > > For reference, prior work on this:
> > > > https://github.com/apache/kafka/pull/11350
> > > > https://github.com/apache/kafka/pull/11432
> > > >
> > > > Best,
> > > >
> > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi Matthiases,
> > > > >
> > > > > I know Scala 2 fairly well, so I'd be happy to review changes that
> add
> > > > > Scala 3 support. However, as Matthias S. said, it has to be driven
> by
> > > > > people who use Scala day-to-day, since I believe most Kafka Streams
> > > > > committers are working with Java.
> > > > >
> > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a
> large
> > > > > undertaking, so option 1 is the first thing we should explore.
> > > > >
> > > > > I don't have any experience with Scala 3 migration topics, but on
> the
> > > > > Scala website it says
> > > > > > The first piece of good news is that the Scala 3 compiler is
> able to
> > > > > read the Scala 2.13 Pickle format and thus it can type check code
> that
> > > > > depends on modules or libraries compiled with Scala 2.13.
> > > > > > One notable example is the Scala 2.13 library. We have indeed
> decided
> > > > > that the Scala 2.13 library is the official standard library for
> Scala
> > > 3.
> > > > > So wouldn't that mean that we are safe in terms of standard library
> > > > > upgrades if we use core_2.13 in the tests?
> > > > >
> > > > > Cheers,
> > > > > Lucas
> > > > >
> > > > >
> > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Thanks for raising this. The `kafka-streams-scala` module seems
> to
> > > be an
> > > > > > important feature for Kafka Streams and I am generally in favor
> of
> > > your
> > > > > > proposal to add Scala 3 support. However, I am personally no
> Scala
> > > > > > person and it sounds like quite some overhead.
> > > > > >
> > > > > > If you are willing to drive and own this initiative happy to
> support
> > > you
> > > > > > to the extend I can.
> > > > > >
> > > > > > About the concrete proposal: my understanding is that :core will
> move
> > > > > > off Scala long-term (not 100% sure what the timeline is, but new
> > > modules
> > > > > > are written in Java only). Thus, down the road the compatibility
> > > issue
> > > > > > would go away naturally, but it's unclear when.
> > > > > >
> > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it
> seems we
> > > > > > could add support for Scala 3 now, taking a risk that it might
> break
> > > in
> > > > > > the future assume that the migration off Scala from core is not
> fast
> > > > > enough.
> > > > > >
> > > > > > For proposal (2), I don't think that it would be easily possible
> for
> > > > > > unit/integration tests. We could fall back to system tests
> though,
> > > but
> > > > > > they would be much more heavy weight of course.
> > > > > >
> > > > > > Might be good to hear from others. We might actually also want
> to do
> > > a
> > > > > > KIP for this?
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > > > > Hey there,
> > > > > > >
> > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala
> > > library.
> > > > > > > Currently, the build system is set up such that
> kafka-streams-scala
> > > > > > > and core (i. e. kafka itself) are compiled with the same Scala
> > > > > > > compiler versions. This is not an optimal situation because it
> > > means
> > > > > > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > > > > > independently of kafka itself. I think this should be changed
> > > > > > >
> > > > > > > The production codebase of scala-streams-kafka actually
> compiles
> > > just
> > > > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes.
> The
> > > > > > > problem is with the tests. These use the `EmbeddedKafkaCluster`
> > > class,
> > > > > > > which means that kafka is pulled into the classpath,
> potentially
> > > > > > > leading to binary compatibility issues.
> > > > > > > I can see several approaches to fixing this:
> > > > > > >
> > > > > > > 1. Run the kafka-streams-scala tests using the compatible
> version
> > > of
> > > > > > > :core if one is available. Currently, this means that
> everything
> > > can
> > > > > > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > > > > kafka-streams-scala_2.13 using core_2.13 and
> kafka-streams-scala_3
> > > > > > > using core_2.13, as these should be compatible), but when a new
> > > > > > > scala-library version is released that is no longer compatible
> with
> > > > > > > 2.13, we won't be able to test that.
> > > > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster,
> instead
> > > > > > > running the test cluster in a separate JVM or perhaps even a
> > > > > > > container.
> > > > > > >
> > > > > > > I'd be willing to get my hands dirty working on this, but
> before I
> > > > > > > start I'd like to get some feedback from the Kafka team
> regarding
> > > the
> > > > > > > approaches outlined above.
> > > > > > >
> > > > > > > All the best
> > > > > > > Matthias Berndt
> > > > >
> > > > KJosep Prat
> > > > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > > > +491715557497 | aiven.io
> > > > Aiven Deutschland GmbH
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.prat@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <
> https://www.facebook.com/aivencloud>
> >   <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Matthias Berndt <ma...@ttmzero.com>.

Thanks Josep, I've applied for a JIRA account and addressed your
review comments.

Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
<jo...@aiven.io.invalid>:
>
> Hi Matthias,
>
> I think for this particular case it would be worth creating a JIRA ticket
> for this as it's a new "feature".
> Regarding the change itself, I think we need to clarify how the release
> process would work. Right now, the script `gradlewAll` is used (which
> basically runs the build with Scala version 2.12 and 2.13). If I understand
> your changes correctly, we would need to run the build 3 times:
> - 1 with property scalaVersion 2.12
> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
>
> I think we should document this and discuss when to have this feature. If I
> remember correctly from when I tried to update Kafka to Scala 3, the idea
> was to push this to a Kafka 4.0 version because we didn't want to maintain
> more than 2 Scala versions at the same time. I would encourage if not
> having a KIP, at least open up a [DISCUSS] thread to clarify some of these
> points.
>
> I'll add some feedback on the PR itself regarding the changes.
>
> Best,
>
> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <ma...@ttmzero.com>
> wrote:
>
> > Hi Matthias J., Hi Lucas, Hi Josep,
> >
> > Thank you for your encouraging responses regarding a Scala 3 port of
> > Kafka-Streams-Scala, and apologies for the late response from my side.
> > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
> > retaining support for 2.13 and 2.12). Almost no changes to the code
> > were required and the tests also pass. Please take a look and let me
> > know what you think :-)
> > https://github.com/apache/kafka/pull/15338
> >
> > All the best
> > Matthias
> >
> > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> > <jo...@aiven.io.invalid>:
> > >
> > > Hi,
> > >
> > > For reference, prior work on this:
> > > https://github.com/apache/kafka/pull/11350
> > > https://github.com/apache/kafka/pull/11432
> > >
> > > Best,
> > >
> > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> > .invalid>
> > > wrote:
> > >
> > > > Hi Matthiases,
> > > >
> > > > I know Scala 2 fairly well, so I'd be happy to review changes that add
> > > > Scala 3 support. However, as Matthias S. said, it has to be driven by
> > > > people who use Scala day-to-day, since I believe most Kafka Streams
> > > > committers are working with Java.
> > > >
> > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large
> > > > undertaking, so option 1 is the first thing we should explore.
> > > >
> > > > I don't have any experience with Scala 3 migration topics, but on the
> > > > Scala website it says
> > > > > The first piece of good news is that the Scala 3 compiler is able to
> > > > read the Scala 2.13 Pickle format and thus it can type check code that
> > > > depends on modules or libraries compiled with Scala 2.13.
> > > > > One notable example is the Scala 2.13 library. We have indeed decided
> > > > that the Scala 2.13 library is the official standard library for Scala
> > 3.
> > > > So wouldn't that mean that we are safe in terms of standard library
> > > > upgrades if we use core_2.13 in the tests?
> > > >
> > > > Cheers,
> > > > Lucas
> > > >
> > > >
> > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > > > >
> > > > > Thanks for raising this. The `kafka-streams-scala` module seems to
> > be an
> > > > > important feature for Kafka Streams and I am generally in favor of
> > your
> > > > > proposal to add Scala 3 support. However, I am personally no Scala
> > > > > person and it sounds like quite some overhead.
> > > > >
> > > > > If you are willing to drive and own this initiative happy to support
> > you
> > > > > to the extend I can.
> > > > >
> > > > > About the concrete proposal: my understanding is that :core will move
> > > > > off Scala long-term (not 100% sure what the timeline is, but new
> > modules
> > > > > are written in Java only). Thus, down the road the compatibility
> > issue
> > > > > would go away naturally, but it's unclear when.
> > > > >
> > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
> > > > > could add support for Scala 3 now, taking a risk that it might break
> > in
> > > > > the future assume that the migration off Scala from core is not fast
> > > > enough.
> > > > >
> > > > > For proposal (2), I don't think that it would be easily possible for
> > > > > unit/integration tests. We could fall back to system tests though,
> > but
> > > > > they would be much more heavy weight of course.
> > > > >
> > > > > Might be good to hear from others. We might actually also want to do
> > a
> > > > > KIP for this?
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > > > Hey there,
> > > > > >
> > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala
> > library.
> > > > > > Currently, the build system is set up such that kafka-streams-scala
> > > > > > and core (i. e. kafka itself) are compiled with the same Scala
> > > > > > compiler versions. This is not an optimal situation because it
> > means
> > > > > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > > > > independently of kafka itself. I think this should be changed
> > > > > >
> > > > > > The production codebase of scala-streams-kafka actually compiles
> > just
> > > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> > > > > > problem is with the tests. These use the `EmbeddedKafkaCluster`
> > class,
> > > > > > which means that kafka is pulled into the classpath, potentially
> > > > > > leading to binary compatibility issues.
> > > > > > I can see several approaches to fixing this:
> > > > > >
> > > > > > 1. Run the kafka-streams-scala tests using the compatible version
> > of
> > > > > > :core if one is available. Currently, this means that everything
> > can
> > > > > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> > > > > > using core_2.13, as these should be compatible), but when a new
> > > > > > scala-library version is released that is no longer compatible with
> > > > > > 2.13, we won't be able to test that.
> > > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> > > > > > running the test cluster in a separate JVM or perhaps even a
> > > > > > container.
> > > > > >
> > > > > > I'd be willing to get my hands dirty working on this, but before I
> > > > > > start I'd like to get some feedback from the Kafka team regarding
> > the
> > > > > > approaches outlined above.
> > > > > >
> > > > > > All the best
> > > > > > Matthias Berndt
> > > >
> > > KJosep Prat
> > > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > > +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
>   <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi Matthias,

I think for this particular case it would be worth creating a JIRA ticket
for this as it's a new "feature".
Regarding the change itself, I think we need to clarify how the release
process would work. Right now, the script `gradlewAll` is used (which
basically runs the build with Scala version 2.12 and 2.13). If I understand
your changes correctly, we would need to run the build 3 times:
- 1 with property scalaVersion 2.12
- 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
- 1 with scalaVersion 2.13 and streamsScalaVersion 3.1

I think we should document this and discuss when to have this feature. If I
remember correctly from when I tried to update Kafka to Scala 3, the idea
was to push this to a Kafka 4.0 version because we didn't want to maintain
more than 2 Scala versions at the same time. I would encourage if not
having a KIP, at least open up a [DISCUSS] thread to clarify some of these
points.

I'll add some feedback on the PR itself regarding the changes.

Best,

On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <ma...@ttmzero.com>
wrote:

> Hi Matthias J., Hi Lucas, Hi Josep,
>
> Thank you for your encouraging responses regarding a Scala 3 port of
> Kafka-Streams-Scala, and apologies for the late response from my side.
> I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
> retaining support for 2.13 and 2.12). Almost no changes to the code
> were required and the tests also pass. Please take a look and let me
> know what you think :-)
> https://github.com/apache/kafka/pull/15338
>
> All the best
> Matthias
>
> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
> <jo...@aiven.io.invalid>:
> >
> > Hi,
> >
> > For reference, prior work on this:
> > https://github.com/apache/kafka/pull/11350
> > https://github.com/apache/kafka/pull/11432
> >
> > Best,
> >
> > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbrutschy@confluent.io
> .invalid>
> > wrote:
> >
> > > Hi Matthiases,
> > >
> > > I know Scala 2 fairly well, so I'd be happy to review changes that add
> > > Scala 3 support. However, as Matthias S. said, it has to be driven by
> > > people who use Scala day-to-day, since I believe most Kafka Streams
> > > committers are working with Java.
> > >
> > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large
> > > undertaking, so option 1 is the first thing we should explore.
> > >
> > > I don't have any experience with Scala 3 migration topics, but on the
> > > Scala website it says
> > > > The first piece of good news is that the Scala 3 compiler is able to
> > > read the Scala 2.13 Pickle format and thus it can type check code that
> > > depends on modules or libraries compiled with Scala 2.13.
> > > > One notable example is the Scala 2.13 library. We have indeed decided
> > > that the Scala 2.13 library is the official standard library for Scala
> 3.
> > > So wouldn't that mean that we are safe in terms of standard library
> > > upgrades if we use core_2.13 in the tests?
> > >
> > > Cheers,
> > > Lucas
> > >
> > >
> > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> > > >
> > > > Thanks for raising this. The `kafka-streams-scala` module seems to
> be an
> > > > important feature for Kafka Streams and I am generally in favor of
> your
> > > > proposal to add Scala 3 support. However, I am personally no Scala
> > > > person and it sounds like quite some overhead.
> > > >
> > > > If you are willing to drive and own this initiative happy to support
> you
> > > > to the extend I can.
> > > >
> > > > About the concrete proposal: my understanding is that :core will move
> > > > off Scala long-term (not 100% sure what the timeline is, but new
> modules
> > > > are written in Java only). Thus, down the road the compatibility
> issue
> > > > would go away naturally, but it's unclear when.
> > > >
> > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
> > > > could add support for Scala 3 now, taking a risk that it might break
> in
> > > > the future assume that the migration off Scala from core is not fast
> > > enough.
> > > >
> > > > For proposal (2), I don't think that it would be easily possible for
> > > > unit/integration tests. We could fall back to system tests though,
> but
> > > > they would be much more heavy weight of course.
> > > >
> > > > Might be good to hear from others. We might actually also want to do
> a
> > > > KIP for this?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > > Hey there,
> > > > >
> > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala
> library.
> > > > > Currently, the build system is set up such that kafka-streams-scala
> > > > > and core (i. e. kafka itself) are compiled with the same Scala
> > > > > compiler versions. This is not an optimal situation because it
> means
> > > > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > > > independently of kafka itself. I think this should be changed
> > > > >
> > > > > The production codebase of scala-streams-kafka actually compiles
> just
> > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> > > > > problem is with the tests. These use the `EmbeddedKafkaCluster`
> class,
> > > > > which means that kafka is pulled into the classpath, potentially
> > > > > leading to binary compatibility issues.
> > > > > I can see several approaches to fixing this:
> > > > >
> > > > > 1. Run the kafka-streams-scala tests using the compatible version
> of
> > > > > :core if one is available. Currently, this means that everything
> can
> > > > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> > > > > using core_2.13, as these should be compatible), but when a new
> > > > > scala-library version is released that is no longer compatible with
> > > > > 2.13, we won't be able to test that.
> > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> > > > > running the test cluster in a separate JVM or perhaps even a
> > > > > container.
> > > > >
> > > > > I'd be willing to get my hands dirty working on this, but before I
> > > > > start I'd like to get some feedback from the Kafka team regarding
> the
> > > > > approaches outlined above.
> > > > >
> > > > > All the best
> > > > > Matthias Berndt
> > >
> > KJosep Prat
> > Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> > +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] <https://www.aiven.io>

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io   |   +491715557497
aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
  <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Matthias Berndt <ma...@ttmzero.com>.

Hi Matthias J., Hi Lucas, Hi Josep,

Thank you for your encouraging responses regarding a Scala 3 port of
Kafka-Streams-Scala, and apologies for the late response from my side.
I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
retaining support for 2.13 and 2.12). Almost no changes to the code
were required and the tests also pass. Please take a look and let me
know what you think :-)
https://github.com/apache/kafka/pull/15338

All the best
Matthias

Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
<jo...@aiven.io.invalid>:
>
> Hi,
>
> For reference, prior work on this:
> https://github.com/apache/kafka/pull/11350
> https://github.com/apache/kafka/pull/11432
>
> Best,
>
> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lb...@confluent.io.invalid>
> wrote:
>
> > Hi Matthiases,
> >
> > I know Scala 2 fairly well, so I'd be happy to review changes that add
> > Scala 3 support. However, as Matthias S. said, it has to be driven by
> > people who use Scala day-to-day, since I believe most Kafka Streams
> > committers are working with Java.
> >
> > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large
> > undertaking, so option 1 is the first thing we should explore.
> >
> > I don't have any experience with Scala 3 migration topics, but on the
> > Scala website it says
> > > The first piece of good news is that the Scala 3 compiler is able to
> > read the Scala 2.13 Pickle format and thus it can type check code that
> > depends on modules or libraries compiled with Scala 2.13.
> > > One notable example is the Scala 2.13 library. We have indeed decided
> > that the Scala 2.13 library is the official standard library for Scala 3.
> > So wouldn't that mean that we are safe in terms of standard library
> > upgrades if we use core_2.13 in the tests?
> >
> > Cheers,
> > Lucas
> >
> >
> > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> wrote:
> > >
> > > Thanks for raising this. The `kafka-streams-scala` module seems to be an
> > > important feature for Kafka Streams and I am generally in favor of your
> > > proposal to add Scala 3 support. However, I am personally no Scala
> > > person and it sounds like quite some overhead.
> > >
> > > If you are willing to drive and own this initiative happy to support you
> > > to the extend I can.
> > >
> > > About the concrete proposal: my understanding is that :core will move
> > > off Scala long-term (not 100% sure what the timeline is, but new modules
> > > are written in Java only). Thus, down the road the compatibility issue
> > > would go away naturally, but it's unclear when.
> > >
> > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
> > > could add support for Scala 3 now, taking a risk that it might break in
> > > the future assume that the migration off Scala from core is not fast
> > enough.
> > >
> > > For proposal (2), I don't think that it would be easily possible for
> > > unit/integration tests. We could fall back to system tests though, but
> > > they would be much more heavy weight of course.
> > >
> > > Might be good to hear from others. We might actually also want to do a
> > > KIP for this?
> > >
> > >
> > > -Matthias
> > >
> > > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > > Hey there,
> > > >
> > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
> > > > Currently, the build system is set up such that kafka-streams-scala
> > > > and core (i. e. kafka itself) are compiled with the same Scala
> > > > compiler versions. This is not an optimal situation because it means
> > > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > > independently of kafka itself. I think this should be changed
> > > >
> > > > The production codebase of scala-streams-kafka actually compiles just
> > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> > > > problem is with the tests. These use the `EmbeddedKafkaCluster` class,
> > > > which means that kafka is pulled into the classpath, potentially
> > > > leading to binary compatibility issues.
> > > > I can see several approaches to fixing this:
> > > >
> > > > 1. Run the kafka-streams-scala tests using the compatible version of
> > > > :core if one is available. Currently, this means that everything can
> > > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> > > > using core_2.13, as these should be compatible), but when a new
> > > > scala-library version is released that is no longer compatible with
> > > > 2.13, we won't be able to test that.
> > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> > > > running the test cluster in a separate JVM or perhaps even a
> > > > container.
> > > >
> > > > I'd be willing to get my hands dirty working on this, but before I
> > > > start I'd like to get some feedback from the Kafka team regarding the
> > > > approaches outlined above.
> > > >
> > > > All the best
> > > > Matthias Berndt
> >
> KJosep Prat
> Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Josep Prat <jo...@aiven.io.INVALID>.

Hi,

For reference, prior work on this:
https://github.com/apache/kafka/pull/11350
https://github.com/apache/kafka/pull/11432

Best,

On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lb...@confluent.io.invalid>
wrote:

> Hi Matthiases,
>
> I know Scala 2 fairly well, so I'd be happy to review changes that add
> Scala 3 support. However, as Matthias S. said, it has to be driven by
> people who use Scala day-to-day, since I believe most Kafka Streams
> committers are working with Java.
>
> Rewriting the tests to not use EmbeddedKafkaCluster seems like a large
> undertaking, so option 1 is the first thing we should explore.
>
> I don't have any experience with Scala 3 migration topics, but on the
> Scala website it says
> > The first piece of good news is that the Scala 3 compiler is able to
> read the Scala 2.13 Pickle format and thus it can type check code that
> depends on modules or libraries compiled with Scala 2.13.
> > One notable example is the Scala 2.13 library. We have indeed decided
> that the Scala 2.13 library is the official standard library for Scala 3.
> So wouldn't that mean that we are safe in terms of standard library
> upgrades if we use core_2.13 in the tests?
>
> Cheers,
> Lucas
>
>
> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> wrote:
> >
> > Thanks for raising this. The `kafka-streams-scala` module seems to be an
> > important feature for Kafka Streams and I am generally in favor of your
> > proposal to add Scala 3 support. However, I am personally no Scala
> > person and it sounds like quite some overhead.
> >
> > If you are willing to drive and own this initiative happy to support you
> > to the extend I can.
> >
> > About the concrete proposal: my understanding is that :core will move
> > off Scala long-term (not 100% sure what the timeline is, but new modules
> > are written in Java only). Thus, down the road the compatibility issue
> > would go away naturally, but it's unclear when.
> >
> > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
> > could add support for Scala 3 now, taking a risk that it might break in
> > the future assume that the migration off Scala from core is not fast
> enough.
> >
> > For proposal (2), I don't think that it would be easily possible for
> > unit/integration tests. We could fall back to system tests though, but
> > they would be much more heavy weight of course.
> >
> > Might be good to hear from others. We might actually also want to do a
> > KIP for this?
> >
> >
> > -Matthias
> >
> > On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > > Hey there,
> > >
> > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
> > > Currently, the build system is set up such that kafka-streams-scala
> > > and core (i. e. kafka itself) are compiled with the same Scala
> > > compiler versions. This is not an optimal situation because it means
> > > that a Scala 3 release of kafka-streams-scala cannot happen
> > > independently of kafka itself. I think this should be changed
> > >
> > > The production codebase of scala-streams-kafka actually compiles just
> > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> > > problem is with the tests. These use the `EmbeddedKafkaCluster` class,
> > > which means that kafka is pulled into the classpath, potentially
> > > leading to binary compatibility issues.
> > > I can see several approaches to fixing this:
> > >
> > > 1. Run the kafka-streams-scala tests using the compatible version of
> > > :core if one is available. Currently, this means that everything can
> > > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> > > using core_2.13, as these should be compatible), but when a new
> > > scala-library version is released that is no longer compatible with
> > > 2.13, we won't be able to test that.
> > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> > > running the test cluster in a separate JVM or perhaps even a
> > > container.
> > >
> > > I'd be willing to get my hands dirty working on this, but before I
> > > start I'd like to get some feedback from the Kafka team regarding the
> > > approaches outlined above.
> > >
> > > All the best
> > > Matthias Berndt
>
KJosep Prat
Open Source Engineering Director, Aivenjosep.prat@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: Kafka-Streams-Scala for Scala 3

Posted by Lucas Brutschy <lb...@confluent.io.INVALID>.

Hi Matthiases,

I know Scala 2 fairly well, so I'd be happy to review changes that add
Scala 3 support. However, as Matthias S. said, it has to be driven by
people who use Scala day-to-day, since I believe most Kafka Streams
committers are working with Java.

Rewriting the tests to not use EmbeddedKafkaCluster seems like a large
undertaking, so option 1 is the first thing we should explore.

I don't have any experience with Scala 3 migration topics, but on the
Scala website it says
> The first piece of good news is that the Scala 3 compiler is able to read the Scala 2.13 Pickle format and thus it can type check code that depends on modules or libraries compiled with Scala 2.13.
> One notable example is the Scala 2.13 library. We have indeed decided that the Scala 2.13 library is the official standard library for Scala 3.
So wouldn't that mean that we are safe in terms of standard library
upgrades if we use core_2.13 in the tests?

Cheers,
Lucas


On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> wrote:
>
> Thanks for raising this. The `kafka-streams-scala` module seems to be an
> important feature for Kafka Streams and I am generally in favor of your
> proposal to add Scala 3 support. However, I am personally no Scala
> person and it sounds like quite some overhead.
>
> If you are willing to drive and own this initiative happy to support you
> to the extend I can.
>
> About the concrete proposal: my understanding is that :core will move
> off Scala long-term (not 100% sure what the timeline is, but new modules
> are written in Java only). Thus, down the road the compatibility issue
> would go away naturally, but it's unclear when.
>
> Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
> could add support for Scala 3 now, taking a risk that it might break in
> the future assume that the migration off Scala from core is not fast enough.
>
> For proposal (2), I don't think that it would be easily possible for
> unit/integration tests. We could fall back to system tests though, but
> they would be much more heavy weight of course.
>
> Might be good to hear from others. We might actually also want to do a
> KIP for this?
>
>
> -Matthias
>
> On 1/20/24 10:34 AM, Matthias Berndt wrote:
> > Hey there,
> >
> > I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
> > Currently, the build system is set up such that kafka-streams-scala
> > and core (i. e. kafka itself) are compiled with the same Scala
> > compiler versions. This is not an optimal situation because it means
> > that a Scala 3 release of kafka-streams-scala cannot happen
> > independently of kafka itself. I think this should be changed
> >
> > The production codebase of scala-streams-kafka actually compiles just
> > fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> > problem is with the tests. These use the `EmbeddedKafkaCluster` class,
> > which means that kafka is pulled into the classpath, potentially
> > leading to binary compatibility issues.
> > I can see several approaches to fixing this:
> >
> > 1. Run the kafka-streams-scala tests using the compatible version of
> > :core if one is available. Currently, this means that everything can
> > be tested (test kafka-streams-scala_2.12 using core_2.12,
> > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> > using core_2.13, as these should be compatible), but when a new
> > scala-library version is released that is no longer compatible with
> > 2.13, we won't be able to test that.
> > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> > running the test cluster in a separate JVM or perhaps even a
> > container.
> >
> > I'd be willing to get my hands dirty working on this, but before I
> > start I'd like to get some feedback from the Kafka team regarding the
> > approaches outlined above.
> >
> > All the best
> > Matthias Berndt

Re: Kafka-Streams-Scala for Scala 3

Posted by "Matthias J. Sax" <mj...@apache.org>.

Thanks for raising this. The `kafka-streams-scala` module seems to be an 
important feature for Kafka Streams and I am generally in favor of your 
proposal to add Scala 3 support. However, I am personally no Scala 
person and it sounds like quite some overhead.

If you are willing to drive and own this initiative happy to support you 
to the extend I can.

About the concrete proposal: my understanding is that :core will move 
off Scala long-term (not 100% sure what the timeline is, but new modules 
are written in Java only). Thus, down the road the compatibility issue 
would go away naturally, but it's unclear when.

Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we 
could add support for Scala 3 now, taking a risk that it might break in 
the future assume that the migration off Scala from core is not fast enough.

For proposal (2), I don't think that it would be easily possible for 
unit/integration tests. We could fall back to system tests though, but 
they would be much more heavy weight of course.

Might be good to hear from others. We might actually also want to do a 
KIP for this?

-Matthias

On 1/20/24 10:34 AM, Matthias Berndt wrote:
> Hey there,
> 
> I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
> Currently, the build system is set up such that kafka-streams-scala
> and core (i. e. kafka itself) are compiled with the same Scala
> compiler versions. This is not an optimal situation because it means
> that a Scala 3 release of kafka-streams-scala cannot happen
> independently of kafka itself. I think this should be changed
> 
> The production codebase of scala-streams-kafka actually compiles just
> fine on Scala 3.3.1 with two lines of trivial syntax changes. The
> problem is with the tests. These use the `EmbeddedKafkaCluster` class,
> which means that kafka is pulled into the classpath, potentially
> leading to binary compatibility issues.
> I can see several approaches to fixing this:
> 
> 1. Run the kafka-streams-scala tests using the compatible version of
> :core if one is available. Currently, this means that everything can
> be tested (test kafka-streams-scala_2.12 using core_2.12,
> kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
> using core_2.13, as these should be compatible), but when a new
> scala-library version is released that is no longer compatible with
> 2.13, we won't be able to test that.
> 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
> running the test cluster in a separate JVM or perhaps even a
> container.
> 
> I'd be willing to get my hands dirty working on this, but before I
> start I'd like to get some feedback from the Kafka team regarding the
> approaches outlined above.
> 
> All the best
> Matthias Berndt