You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Josep Prat <jo...@aiven.io.INVALID> on 2021/10/08 13:07:07 UTC
[DISCUSS] Bringing Kafka towards Scala 3
Hi there,
For the last month I was actively working on migrating Apache Kafka to
Scala 3. For what I could see, Apache Kafka is the biggest mixed Java-Scala
project to attempt a migration to Scala 3. This means, a lot of regressions
in regards to Java and Scala interoperability at bytecode were found within
this task. You can take a look at the PR here:
https://github.com/apache/kafka/pull/11350
Currently, the status of the migration is as follows:
- All tests pass in Scala 3 except one (
KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
Root cause for this is this bug with no easy workaround: Type erased for
by-name parameters lampepfl/dotty#13638
<https://github.com/lampepfl/dotty/issues/13638>
- All tests pass in Scala 2
- It still uses a snapshot build of Gradle with support for Scala 3
- Spotbugs detected some new errors when the code was compiled with
Scala 3. Some are simply false positives, but a small subset seem like they
might be legit bugs in Scala 3. I still need to come up with a minimized
reproducer for those cases.
As one can see, most of the changes are arguably low-level and no big
rewrite was needed. The biggest refactoring was splitting several classes
into their own file.
Changes made in this PR could be categorized in 3 different groups:
a) Needed changes because of changes in syntax or compiler being currently
more strict
b) Changes where a workaround is present but they are (or will be) fixed in
upcoming Scala versions
c) Changes in the build file
In order to have a smooth and manageable migration to Scala 3, I propose
perform the migration in several steps:
- Firstly apply all changes that would fall into category *a)*
- Once a new Scala version that incorporates the mentioned fixes is
released, revisit point changes in point *b)*
- Once Type erased for by-name parameters lampepfl/dotty#13638
<https://github.com/lampepfl/dotty/issues/13638> is resolved or a
workaround is found and Gradle with Scala 3 support is released we could
tackle the final step and incorporate changes mentioned in point *c)*
- After this, we would run the Scala Migration Tool to automatically
rewrite the code that uses removed syntax or language constructs. (See
https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
)
What do you think about this approach? Please feel free to share your
opinion on the proposed plan.
Also, I'm not sure if this should be written in a KIP form or not, no
public interfaces needed to be modified.
For extra information, these are the bugs reported to Scala 3 so far:
- https://github.com/lampepfl/dotty/issues/13549
- https://github.com/lampepfl/dotty/issues/13572
- https://github.com/lampepfl/dotty/issues/13630
- https://github.com/lampepfl/dotty/issues/13638
- https://github.com/lampepfl/dotty/issues/13645
Thank you in advance!
--
Josep Prat
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491715557497
*w:* aiven.io
*e:* josep.prat@aiven.io
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Josep Prat <jo...@aiven.io.INVALID>.
Also, on a related note, the Scala Center was interested in adding Kafka to
the Community Build once the migration is done. See
https://gitter.im/scala/contributors?at=6154a73f29ddcd0293f5e01e
It turns out, Kafka is a good stress test for Scala/Java interoperability.
For Kafka, and other similar Scala/Java projects it would mean that new
Scala versions won't break at compatibility level between versions.
Best,
———
Josep Prat
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491715557497
w: aiven.io
e: josep.prat@aiven.io
On Fri, Oct 8, 2021, 16:35 Josep Prat <jo...@aiven.io> wrote:
> Hi Ismael,
>
> Thanks for the reply. I don't think the demand is high at the moment, but
> it will increase overtime.
> What do you think about me submitting a PR now with only the changes I
> mention in group a)? They are small changes like dropping extra
> parenthesis, or stopping shadowing names. The resulting code would be
> cleaner Scala that can cross compile.
>
> Thanks again,
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.prat@aiven.io
>
> On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
>
>> Hi Josep,
>>
>> Thanks for looking into this. As you said, it seems like there are a
>> number
>> of rough edges still. Do we have many users asking for this right now? If
>> not, it may make sense to wait for things to get a bit more mature before
>> adding the burden of a third Scala version. Ideally, we would drop Scala
>> 2.12 before we add Scala 3 support.
>>
>> Ismael
>>
>> On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <jo...@aiven.io.invalid>
>> wrote:
>>
>> > Hi there,
>> > For the last month I was actively working on migrating Apache Kafka to
>> > Scala 3. For what I could see, Apache Kafka is the biggest mixed
>> Java-Scala
>> > project to attempt a migration to Scala 3. This means, a lot of
>> regressions
>> > in regards to Java and Scala interoperability at bytecode were found
>> within
>> > this task. You can take a look at the PR here:
>> > https://github.com/apache/kafka/pull/11350
>> > Currently, the status of the migration is as follows:
>> >
>> > - All tests pass in Scala 3 except one (
>> >
>> >
>> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
>> > Root cause for this is this bug with no easy workaround: Type erased
>> for
>> > by-name parameters lampepfl/dotty#13638
>> > <https://github.com/lampepfl/dotty/issues/13638>
>> > - All tests pass in Scala 2
>> > - It still uses a snapshot build of Gradle with support for Scala 3
>> > - Spotbugs detected some new errors when the code was compiled with
>> > Scala 3. Some are simply false positives, but a small subset seem
>> like
>> > they
>> > might be legit bugs in Scala 3. I still need to come up with a
>> minimized
>> > reproducer for those cases.
>> >
>> > As one can see, most of the changes are arguably low-level and no big
>> > rewrite was needed. The biggest refactoring was splitting several
>> classes
>> > into their own file.
>> >
>> > Changes made in this PR could be categorized in 3 different groups:
>> > a) Needed changes because of changes in syntax or compiler being
>> currently
>> > more strict
>> > b) Changes where a workaround is present but they are (or will be)
>> fixed in
>> > upcoming Scala versions
>> > c) Changes in the build file
>> >
>> > In order to have a smooth and manageable migration to Scala 3, I propose
>> > perform the migration in several steps:
>> >
>> > - Firstly apply all changes that would fall into category *a)*
>> > - Once a new Scala version that incorporates the mentioned fixes is
>> > released, revisit point changes in point *b)*
>> > - Once Type erased for by-name parameters lampepfl/dotty#13638
>> > <https://github.com/lampepfl/dotty/issues/13638> is resolved or a
>> > workaround is found and Gradle with Scala 3 support is released we
>> could
>> > tackle the final step and incorporate changes mentioned in point *c)*
>> > - After this, we would run the Scala Migration Tool to automatically
>> > rewrite the code that uses removed syntax or language constructs.
>> (See
>> >
>> >
>> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
>> > )
>> >
>> >
>> > What do you think about this approach? Please feel free to share your
>> > opinion on the proposed plan.
>> > Also, I'm not sure if this should be written in a KIP form or not, no
>> > public interfaces needed to be modified.
>> >
>> >
>> > For extra information, these are the bugs reported to Scala 3 so far:
>> >
>> > - https://github.com/lampepfl/dotty/issues/13549
>> > - https://github.com/lampepfl/dotty/issues/13572
>> > - https://github.com/lampepfl/dotty/issues/13630
>> > - https://github.com/lampepfl/dotty/issues/13638
>> > - https://github.com/lampepfl/dotty/issues/13645
>> >
>> >
>> > Thank you in advance!
>> > --
>> >
>> > Josep Prat
>> >
>> > *Aiven Deutschland GmbH*
>> >
>> > Immanuelkirchstraße 26, 10405 Berlin
>> >
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >
>> > *m:* +491715557497
>> >
>> > *w:* aiven.io
>> >
>> > *e:* josep.prat@aiven.io
>> >
>>
>
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Josep Prat <jo...@aiven.io.INVALID>.
Hi there,
I just wanted to bump this thread as there has been a great development.
Gradle 7.3 is now released and it comes with Scala 3 support (release notes
<https://docs.gradle.org/7.3/release-notes.html>). This means that one of
the roadblocks to migrate Kafka to Scala 3 has been cleared.
Shall I write a KIP to decide if we want to migrate to Scala 3 right away
or wait until Kafka 4.0 in which Scala 2.12 will be removed?
I also had a favour to ask, could someone take a look at
https://github.com/apache/kafka/pull/11432? The purpose of the PR is to
improve some Scala code to make it easy to migrate in the future, the PR
has been sitting there for several weeks now. I would highly appreciate
some reviews in there so we can move it forward before more merge conflicts
appear.
Thanks a lot!
On Tue, Nov 2, 2021 at 9:06 AM Josep Prat <jo...@aiven.io> wrote:
> Hi Colin,
>
> We can take 2 different paths, both of them need a similar effort.
> — Offer Scala 3 along with Scala 2.12 and 2.13
> — Offer Scala 3 only once Scala 2.12 is removed
>
> On the draft PR I shared in the initial email (
> https://github.com/apache/kafka/pull/11350), I used the first approach,
> and it didn't require any additional work. This means we can decide to take
> any of the 2 approaches freely. Offering Scala 3 only when 2.13 is removed
> means to wait for a bit, but it's perfectly doable.
> My personal.opinion is that, given Scala 3 is a major release (epoch
> according to Scala version schema), we might want to build in Scala 3
> sooner rather than later. We could label this as experimental Scala 3
> support.
> However, until Gradle 7.3 is released, we can't proceed any further with
> building with Scala 3. Currently, Gradle 7.3 is under RC3, so I expect it
> to be available soon.
>
> Regarding the changes I propose to add right now as I mentioned in the
> first email, you can see exactly what I meant in this PR:
> https://github.com/apache/kafka/pull/11432. Sometimes, Kafka uses some
> syntax or approaches that are deprecated and they are removed in Scala 3.
> This PR goes over those and uses the "accepted" syntax. Feel free to take
> a look and review it.
>
> Thanks for your feedback Colin!
>
> Best,
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.prat@aiven.io
>
> On Mon, Nov 1, 2021, 23:45 Colin McCabe <cm...@apache.org> wrote:
>
>> Thanks for looking at this, Josep... I guess the way I always imagined
>> this happening was that a Scala 3 release would become one of our two
>> supported Scala versions. So instead of 2.12 and 2.13, we'd have 2.13 and
>> 3.x, for some X. Do you think that approach would be practical?
>>
>> best,
>> Colin
>>
>>
>> On Fri, Oct 8, 2021, at 09:05, Ismael Juma wrote:
>> > Yeah, changes that are good generally can be submitted any time.
>> >
>> > Ismael
>> >
>> > On Fri, Oct 8, 2021 at 7:35 AM Josep Prat <jo...@aiven.io.invalid>
>> > wrote:
>> >
>> >> Hi Ismael,
>> >>
>> >> Thanks for the reply. I don't think the demand is high at the moment,
>> but
>> >> it will increase overtime.
>> >> What do you think about me submitting a PR now with only the changes I
>> >> mention in group a)? They are small changes like dropping extra
>> >> parenthesis, or stopping shadowing names. The resulting code would be
>> >> cleaner Scala that can cross compile.
>> >>
>> >> Thanks again,
>> >> ———
>> >> Josep Prat
>> >>
>> >> Aiven Deutschland GmbH
>> >>
>> >> Immanuelkirchstraße 26, 10405 Berlin
>> >>
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >>
>> >> m: +491715557497
>> >>
>> >> w: aiven.io
>> >>
>> >> e: josep.prat@aiven.io
>> >>
>> >> On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
>> >>
>> >> > Hi Josep,
>> >> >
>> >> > Thanks for looking into this. As you said, it seems like there are a
>> >> number
>> >> > of rough edges still. Do we have many users asking for this right
>> now? If
>> >> > not, it may make sense to wait for things to get a bit more mature
>> before
>> >> > adding the burden of a third Scala version. Ideally, we would drop
>> Scala
>> >> > 2.12 before we add Scala 3 support.
>> >> >
>> >> > Ismael
>> >> >
>> >> > On Fri, Oct 8, 2021 at 6:07 AM Josep Prat
>> <jo...@aiven.io.invalid>
>> >> > wrote:
>> >> >
>> >> > > Hi there,
>> >> > > For the last month I was actively working on migrating Apache
>> Kafka to
>> >> > > Scala 3. For what I could see, Apache Kafka is the biggest mixed
>> >> > Java-Scala
>> >> > > project to attempt a migration to Scala 3. This means, a lot of
>> >> > regressions
>> >> > > in regards to Java and Scala interoperability at bytecode were
>> found
>> >> > within
>> >> > > this task. You can take a look at the PR here:
>> >> > > https://github.com/apache/kafka/pull/11350
>> >> > > Currently, the status of the migration is as follows:
>> >> > >
>> >> > > - All tests pass in Scala 3 except one (
>> >> > >
>> >> > >
>> >> >
>> >>
>> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
>> >> > > Root cause for this is this bug with no easy workaround: Type
>> erased
>> >> > for
>> >> > > by-name parameters lampepfl/dotty#13638
>> >> > > <https://github.com/lampepfl/dotty/issues/13638>
>> >> > > - All tests pass in Scala 2
>> >> > > - It still uses a snapshot build of Gradle with support for
>> Scala 3
>> >> > > - Spotbugs detected some new errors when the code was compiled
>> with
>> >> > > Scala 3. Some are simply false positives, but a small subset
>> seem
>> >> like
>> >> > > they
>> >> > > might be legit bugs in Scala 3. I still need to come up with a
>> >> > minimized
>> >> > > reproducer for those cases.
>> >> > >
>> >> > > As one can see, most of the changes are arguably low-level and no
>> big
>> >> > > rewrite was needed. The biggest refactoring was splitting several
>> >> classes
>> >> > > into their own file.
>> >> > >
>> >> > > Changes made in this PR could be categorized in 3 different groups:
>> >> > > a) Needed changes because of changes in syntax or compiler being
>> >> > currently
>> >> > > more strict
>> >> > > b) Changes where a workaround is present but they are (or will be)
>> >> fixed
>> >> > in
>> >> > > upcoming Scala versions
>> >> > > c) Changes in the build file
>> >> > >
>> >> > > In order to have a smooth and manageable migration to Scala 3, I
>> >> propose
>> >> > > perform the migration in several steps:
>> >> > >
>> >> > > - Firstly apply all changes that would fall into category *a)*
>> >> > > - Once a new Scala version that incorporates the mentioned
>> fixes is
>> >> > > released, revisit point changes in point *b)*
>> >> > > - Once Type erased for by-name parameters lampepfl/dotty#13638
>> >> > > <https://github.com/lampepfl/dotty/issues/13638> is resolved
>> or a
>> >> > > workaround is found and Gradle with Scala 3 support is released
>> we
>> >> > could
>> >> > > tackle the final step and incorporate changes mentioned in point
>> >> *c)*
>> >> > > - After this, we would run the Scala Migration Tool to
>> automatically
>> >> > > rewrite the code that uses removed syntax or language
>> constructs.
>> >> (See
>> >> > >
>> >> > >
>> >> >
>> >>
>> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
>> >> > > )
>> >> > >
>> >> > >
>> >> > > What do you think about this approach? Please feel free to share
>> your
>> >> > > opinion on the proposed plan.
>> >> > > Also, I'm not sure if this should be written in a KIP form or not,
>> no
>> >> > > public interfaces needed to be modified.
>> >> > >
>> >> > >
>> >> > > For extra information, these are the bugs reported to Scala 3 so
>> far:
>> >> > >
>> >> > > - https://github.com/lampepfl/dotty/issues/13549
>> >> > > - https://github.com/lampepfl/dotty/issues/13572
>> >> > > - https://github.com/lampepfl/dotty/issues/13630
>> >> > > - https://github.com/lampepfl/dotty/issues/13638
>> >> > > - https://github.com/lampepfl/dotty/issues/13645
>> >> > >
>> >> > >
>> >> > > Thank you in advance!
>> >> > > --
>> >> > >
>> >> > > Josep Prat
>> >> > >
>> >> > > *Aiven Deutschland GmbH*
>> >> > >
>> >> > > Immanuelkirchstraße 26, 10405 Berlin
>> >> > >
>> >> > > Amtsgericht Charlottenburg, HRB 209739 B
>> >> > >
>> >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >> > >
>> >> > > *m:* +491715557497
>> >> > >
>> >> > > *w:* aiven.io
>> >> > >
>> >> > > *e:* josep.prat@aiven.io
>> >> > >
>> >> >
>> >>
>>
>
--
Josep Prat
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491715557497
*w:* aiven.io
*e:* josep.prat@aiven.io
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Josep Prat <jo...@aiven.io.INVALID>.
Hi Colin,
We can take 2 different paths, both of them need a similar effort.
— Offer Scala 3 along with Scala 2.12 and 2.13
— Offer Scala 3 only once Scala 2.12 is removed
On the draft PR I shared in the initial email (
https://github.com/apache/kafka/pull/11350), I used the first approach, and
it didn't require any additional work. This means we can decide to take any
of the 2 approaches freely. Offering Scala 3 only when 2.13 is removed
means to wait for a bit, but it's perfectly doable.
My personal.opinion is that, given Scala 3 is a major release (epoch
according to Scala version schema), we might want to build in Scala 3
sooner rather than later. We could label this as experimental Scala 3
support.
However, until Gradle 7.3 is released, we can't proceed any further with
building with Scala 3. Currently, Gradle 7.3 is under RC3, so I expect it
to be available soon.
Regarding the changes I propose to add right now as I mentioned in the
first email, you can see exactly what I meant in this PR:
https://github.com/apache/kafka/pull/11432. Sometimes, Kafka uses some
syntax or approaches that are deprecated and they are removed in Scala 3.
This PR goes over those and uses the "accepted" syntax. Feel free to take a
look and review it.
Thanks for your feedback Colin!
Best,
———
Josep Prat
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491715557497
w: aiven.io
e: josep.prat@aiven.io
On Mon, Nov 1, 2021, 23:45 Colin McCabe <cm...@apache.org> wrote:
> Thanks for looking at this, Josep... I guess the way I always imagined
> this happening was that a Scala 3 release would become one of our two
> supported Scala versions. So instead of 2.12 and 2.13, we'd have 2.13 and
> 3.x, for some X. Do you think that approach would be practical?
>
> best,
> Colin
>
>
> On Fri, Oct 8, 2021, at 09:05, Ismael Juma wrote:
> > Yeah, changes that are good generally can be submitted any time.
> >
> > Ismael
> >
> > On Fri, Oct 8, 2021 at 7:35 AM Josep Prat <jo...@aiven.io.invalid>
> > wrote:
> >
> >> Hi Ismael,
> >>
> >> Thanks for the reply. I don't think the demand is high at the moment,
> but
> >> it will increase overtime.
> >> What do you think about me submitting a PR now with only the changes I
> >> mention in group a)? They are small changes like dropping extra
> >> parenthesis, or stopping shadowing names. The resulting code would be
> >> cleaner Scala that can cross compile.
> >>
> >> Thanks again,
> >> ———
> >> Josep Prat
> >>
> >> Aiven Deutschland GmbH
> >>
> >> Immanuelkirchstraße 26, 10405 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> m: +491715557497
> >>
> >> w: aiven.io
> >>
> >> e: josep.prat@aiven.io
> >>
> >> On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
> >>
> >> > Hi Josep,
> >> >
> >> > Thanks for looking into this. As you said, it seems like there are a
> >> number
> >> > of rough edges still. Do we have many users asking for this right
> now? If
> >> > not, it may make sense to wait for things to get a bit more mature
> before
> >> > adding the burden of a third Scala version. Ideally, we would drop
> Scala
> >> > 2.12 before we add Scala 3 support.
> >> >
> >> > Ismael
> >> >
> >> > On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <josep.prat@aiven.io.invalid
> >
> >> > wrote:
> >> >
> >> > > Hi there,
> >> > > For the last month I was actively working on migrating Apache Kafka
> to
> >> > > Scala 3. For what I could see, Apache Kafka is the biggest mixed
> >> > Java-Scala
> >> > > project to attempt a migration to Scala 3. This means, a lot of
> >> > regressions
> >> > > in regards to Java and Scala interoperability at bytecode were found
> >> > within
> >> > > this task. You can take a look at the PR here:
> >> > > https://github.com/apache/kafka/pull/11350
> >> > > Currently, the status of the migration is as follows:
> >> > >
> >> > > - All tests pass in Scala 3 except one (
> >> > >
> >> > >
> >> >
> >>
> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
> >> > > Root cause for this is this bug with no easy workaround: Type
> erased
> >> > for
> >> > > by-name parameters lampepfl/dotty#13638
> >> > > <https://github.com/lampepfl/dotty/issues/13638>
> >> > > - All tests pass in Scala 2
> >> > > - It still uses a snapshot build of Gradle with support for
> Scala 3
> >> > > - Spotbugs detected some new errors when the code was compiled
> with
> >> > > Scala 3. Some are simply false positives, but a small subset seem
> >> like
> >> > > they
> >> > > might be legit bugs in Scala 3. I still need to come up with a
> >> > minimized
> >> > > reproducer for those cases.
> >> > >
> >> > > As one can see, most of the changes are arguably low-level and no
> big
> >> > > rewrite was needed. The biggest refactoring was splitting several
> >> classes
> >> > > into their own file.
> >> > >
> >> > > Changes made in this PR could be categorized in 3 different groups:
> >> > > a) Needed changes because of changes in syntax or compiler being
> >> > currently
> >> > > more strict
> >> > > b) Changes where a workaround is present but they are (or will be)
> >> fixed
> >> > in
> >> > > upcoming Scala versions
> >> > > c) Changes in the build file
> >> > >
> >> > > In order to have a smooth and manageable migration to Scala 3, I
> >> propose
> >> > > perform the migration in several steps:
> >> > >
> >> > > - Firstly apply all changes that would fall into category *a)*
> >> > > - Once a new Scala version that incorporates the mentioned fixes
> is
> >> > > released, revisit point changes in point *b)*
> >> > > - Once Type erased for by-name parameters lampepfl/dotty#13638
> >> > > <https://github.com/lampepfl/dotty/issues/13638> is resolved or
> a
> >> > > workaround is found and Gradle with Scala 3 support is released
> we
> >> > could
> >> > > tackle the final step and incorporate changes mentioned in point
> >> *c)*
> >> > > - After this, we would run the Scala Migration Tool to
> automatically
> >> > > rewrite the code that uses removed syntax or language constructs.
> >> (See
> >> > >
> >> > >
> >> >
> >>
> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
> >> > > )
> >> > >
> >> > >
> >> > > What do you think about this approach? Please feel free to share
> your
> >> > > opinion on the proposed plan.
> >> > > Also, I'm not sure if this should be written in a KIP form or not,
> no
> >> > > public interfaces needed to be modified.
> >> > >
> >> > >
> >> > > For extra information, these are the bugs reported to Scala 3 so
> far:
> >> > >
> >> > > - https://github.com/lampepfl/dotty/issues/13549
> >> > > - https://github.com/lampepfl/dotty/issues/13572
> >> > > - https://github.com/lampepfl/dotty/issues/13630
> >> > > - https://github.com/lampepfl/dotty/issues/13638
> >> > > - https://github.com/lampepfl/dotty/issues/13645
> >> > >
> >> > >
> >> > > Thank you in advance!
> >> > > --
> >> > >
> >> > > Josep Prat
> >> > >
> >> > > *Aiven Deutschland GmbH*
> >> > >
> >> > > Immanuelkirchstraße 26, 10405 Berlin
> >> > >
> >> > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > >
> >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > >
> >> > > *m:* +491715557497
> >> > >
> >> > > *w:* aiven.io
> >> > >
> >> > > *e:* josep.prat@aiven.io
> >> > >
> >> >
> >>
>
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Colin McCabe <cm...@apache.org>.
Thanks for looking at this, Josep... I guess the way I always imagined this happening was that a Scala 3 release would become one of our two supported Scala versions. So instead of 2.12 and 2.13, we'd have 2.13 and 3.x, for some X. Do you think that approach would be practical?
best,
Colin
On Fri, Oct 8, 2021, at 09:05, Ismael Juma wrote:
> Yeah, changes that are good generally can be submitted any time.
>
> Ismael
>
> On Fri, Oct 8, 2021 at 7:35 AM Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
>> Hi Ismael,
>>
>> Thanks for the reply. I don't think the demand is high at the moment, but
>> it will increase overtime.
>> What do you think about me submitting a PR now with only the changes I
>> mention in group a)? They are small changes like dropping extra
>> parenthesis, or stopping shadowing names. The resulting code would be
>> cleaner Scala that can cross compile.
>>
>> Thanks again,
>> ———
>> Josep Prat
>>
>> Aiven Deutschland GmbH
>>
>> Immanuelkirchstraße 26, 10405 Berlin
>>
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>
>> m: +491715557497
>>
>> w: aiven.io
>>
>> e: josep.prat@aiven.io
>>
>> On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
>>
>> > Hi Josep,
>> >
>> > Thanks for looking into this. As you said, it seems like there are a
>> number
>> > of rough edges still. Do we have many users asking for this right now? If
>> > not, it may make sense to wait for things to get a bit more mature before
>> > adding the burden of a third Scala version. Ideally, we would drop Scala
>> > 2.12 before we add Scala 3 support.
>> >
>> > Ismael
>> >
>> > On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <jo...@aiven.io.invalid>
>> > wrote:
>> >
>> > > Hi there,
>> > > For the last month I was actively working on migrating Apache Kafka to
>> > > Scala 3. For what I could see, Apache Kafka is the biggest mixed
>> > Java-Scala
>> > > project to attempt a migration to Scala 3. This means, a lot of
>> > regressions
>> > > in regards to Java and Scala interoperability at bytecode were found
>> > within
>> > > this task. You can take a look at the PR here:
>> > > https://github.com/apache/kafka/pull/11350
>> > > Currently, the status of the migration is as follows:
>> > >
>> > > - All tests pass in Scala 3 except one (
>> > >
>> > >
>> >
>> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
>> > > Root cause for this is this bug with no easy workaround: Type erased
>> > for
>> > > by-name parameters lampepfl/dotty#13638
>> > > <https://github.com/lampepfl/dotty/issues/13638>
>> > > - All tests pass in Scala 2
>> > > - It still uses a snapshot build of Gradle with support for Scala 3
>> > > - Spotbugs detected some new errors when the code was compiled with
>> > > Scala 3. Some are simply false positives, but a small subset seem
>> like
>> > > they
>> > > might be legit bugs in Scala 3. I still need to come up with a
>> > minimized
>> > > reproducer for those cases.
>> > >
>> > > As one can see, most of the changes are arguably low-level and no big
>> > > rewrite was needed. The biggest refactoring was splitting several
>> classes
>> > > into their own file.
>> > >
>> > > Changes made in this PR could be categorized in 3 different groups:
>> > > a) Needed changes because of changes in syntax or compiler being
>> > currently
>> > > more strict
>> > > b) Changes where a workaround is present but they are (or will be)
>> fixed
>> > in
>> > > upcoming Scala versions
>> > > c) Changes in the build file
>> > >
>> > > In order to have a smooth and manageable migration to Scala 3, I
>> propose
>> > > perform the migration in several steps:
>> > >
>> > > - Firstly apply all changes that would fall into category *a)*
>> > > - Once a new Scala version that incorporates the mentioned fixes is
>> > > released, revisit point changes in point *b)*
>> > > - Once Type erased for by-name parameters lampepfl/dotty#13638
>> > > <https://github.com/lampepfl/dotty/issues/13638> is resolved or a
>> > > workaround is found and Gradle with Scala 3 support is released we
>> > could
>> > > tackle the final step and incorporate changes mentioned in point
>> *c)*
>> > > - After this, we would run the Scala Migration Tool to automatically
>> > > rewrite the code that uses removed syntax or language constructs.
>> (See
>> > >
>> > >
>> >
>> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
>> > > )
>> > >
>> > >
>> > > What do you think about this approach? Please feel free to share your
>> > > opinion on the proposed plan.
>> > > Also, I'm not sure if this should be written in a KIP form or not, no
>> > > public interfaces needed to be modified.
>> > >
>> > >
>> > > For extra information, these are the bugs reported to Scala 3 so far:
>> > >
>> > > - https://github.com/lampepfl/dotty/issues/13549
>> > > - https://github.com/lampepfl/dotty/issues/13572
>> > > - https://github.com/lampepfl/dotty/issues/13630
>> > > - https://github.com/lampepfl/dotty/issues/13638
>> > > - https://github.com/lampepfl/dotty/issues/13645
>> > >
>> > >
>> > > Thank you in advance!
>> > > --
>> > >
>> > > Josep Prat
>> > >
>> > > *Aiven Deutschland GmbH*
>> > >
>> > > Immanuelkirchstraße 26, 10405 Berlin
>> > >
>> > > Amtsgericht Charlottenburg, HRB 209739 B
>> > >
>> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >
>> > > *m:* +491715557497
>> > >
>> > > *w:* aiven.io
>> > >
>> > > *e:* josep.prat@aiven.io
>> > >
>> >
>>
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Ismael Juma <is...@juma.me.uk>.
Yeah, changes that are good generally can be submitted any time.
Ismael
On Fri, Oct 8, 2021 at 7:35 AM Josep Prat <jo...@aiven.io.invalid>
wrote:
> Hi Ismael,
>
> Thanks for the reply. I don't think the demand is high at the moment, but
> it will increase overtime.
> What do you think about me submitting a PR now with only the changes I
> mention in group a)? They are small changes like dropping extra
> parenthesis, or stopping shadowing names. The resulting code would be
> cleaner Scala that can cross compile.
>
> Thanks again,
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.prat@aiven.io
>
> On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Josep,
> >
> > Thanks for looking into this. As you said, it seems like there are a
> number
> > of rough edges still. Do we have many users asking for this right now? If
> > not, it may make sense to wait for things to get a bit more mature before
> > adding the burden of a third Scala version. Ideally, we would drop Scala
> > 2.12 before we add Scala 3 support.
> >
> > Ismael
> >
> > On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <jo...@aiven.io.invalid>
> > wrote:
> >
> > > Hi there,
> > > For the last month I was actively working on migrating Apache Kafka to
> > > Scala 3. For what I could see, Apache Kafka is the biggest mixed
> > Java-Scala
> > > project to attempt a migration to Scala 3. This means, a lot of
> > regressions
> > > in regards to Java and Scala interoperability at bytecode were found
> > within
> > > this task. You can take a look at the PR here:
> > > https://github.com/apache/kafka/pull/11350
> > > Currently, the status of the migration is as follows:
> > >
> > > - All tests pass in Scala 3 except one (
> > >
> > >
> >
> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
> > > Root cause for this is this bug with no easy workaround: Type erased
> > for
> > > by-name parameters lampepfl/dotty#13638
> > > <https://github.com/lampepfl/dotty/issues/13638>
> > > - All tests pass in Scala 2
> > > - It still uses a snapshot build of Gradle with support for Scala 3
> > > - Spotbugs detected some new errors when the code was compiled with
> > > Scala 3. Some are simply false positives, but a small subset seem
> like
> > > they
> > > might be legit bugs in Scala 3. I still need to come up with a
> > minimized
> > > reproducer for those cases.
> > >
> > > As one can see, most of the changes are arguably low-level and no big
> > > rewrite was needed. The biggest refactoring was splitting several
> classes
> > > into their own file.
> > >
> > > Changes made in this PR could be categorized in 3 different groups:
> > > a) Needed changes because of changes in syntax or compiler being
> > currently
> > > more strict
> > > b) Changes where a workaround is present but they are (or will be)
> fixed
> > in
> > > upcoming Scala versions
> > > c) Changes in the build file
> > >
> > > In order to have a smooth and manageable migration to Scala 3, I
> propose
> > > perform the migration in several steps:
> > >
> > > - Firstly apply all changes that would fall into category *a)*
> > > - Once a new Scala version that incorporates the mentioned fixes is
> > > released, revisit point changes in point *b)*
> > > - Once Type erased for by-name parameters lampepfl/dotty#13638
> > > <https://github.com/lampepfl/dotty/issues/13638> is resolved or a
> > > workaround is found and Gradle with Scala 3 support is released we
> > could
> > > tackle the final step and incorporate changes mentioned in point
> *c)*
> > > - After this, we would run the Scala Migration Tool to automatically
> > > rewrite the code that uses removed syntax or language constructs.
> (See
> > >
> > >
> >
> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
> > > )
> > >
> > >
> > > What do you think about this approach? Please feel free to share your
> > > opinion on the proposed plan.
> > > Also, I'm not sure if this should be written in a KIP form or not, no
> > > public interfaces needed to be modified.
> > >
> > >
> > > For extra information, these are the bugs reported to Scala 3 so far:
> > >
> > > - https://github.com/lampepfl/dotty/issues/13549
> > > - https://github.com/lampepfl/dotty/issues/13572
> > > - https://github.com/lampepfl/dotty/issues/13630
> > > - https://github.com/lampepfl/dotty/issues/13638
> > > - https://github.com/lampepfl/dotty/issues/13645
> > >
> > >
> > > Thank you in advance!
> > > --
> > >
> > > Josep Prat
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491715557497
> > >
> > > *w:* aiven.io
> > >
> > > *e:* josep.prat@aiven.io
> > >
> >
>
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Josep Prat <jo...@aiven.io.INVALID>.
Hi Ismael,
Thanks for the reply. I don't think the demand is high at the moment, but
it will increase overtime.
What do you think about me submitting a PR now with only the changes I
mention in group a)? They are small changes like dropping extra
parenthesis, or stopping shadowing names. The resulting code would be
cleaner Scala that can cross compile.
Thanks again,
———
Josep Prat
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491715557497
w: aiven.io
e: josep.prat@aiven.io
On Fri, Oct 8, 2021, 16:22 Ismael Juma <is...@juma.me.uk> wrote:
> Hi Josep,
>
> Thanks for looking into this. As you said, it seems like there are a number
> of rough edges still. Do we have many users asking for this right now? If
> not, it may make sense to wait for things to get a bit more mature before
> adding the burden of a third Scala version. Ideally, we would drop Scala
> 2.12 before we add Scala 3 support.
>
> Ismael
>
> On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <jo...@aiven.io.invalid>
> wrote:
>
> > Hi there,
> > For the last month I was actively working on migrating Apache Kafka to
> > Scala 3. For what I could see, Apache Kafka is the biggest mixed
> Java-Scala
> > project to attempt a migration to Scala 3. This means, a lot of
> regressions
> > in regards to Java and Scala interoperability at bytecode were found
> within
> > this task. You can take a look at the PR here:
> > https://github.com/apache/kafka/pull/11350
> > Currently, the status of the migration is as follows:
> >
> > - All tests pass in Scala 3 except one (
> >
> >
> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
> > Root cause for this is this bug with no easy workaround: Type erased
> for
> > by-name parameters lampepfl/dotty#13638
> > <https://github.com/lampepfl/dotty/issues/13638>
> > - All tests pass in Scala 2
> > - It still uses a snapshot build of Gradle with support for Scala 3
> > - Spotbugs detected some new errors when the code was compiled with
> > Scala 3. Some are simply false positives, but a small subset seem like
> > they
> > might be legit bugs in Scala 3. I still need to come up with a
> minimized
> > reproducer for those cases.
> >
> > As one can see, most of the changes are arguably low-level and no big
> > rewrite was needed. The biggest refactoring was splitting several classes
> > into their own file.
> >
> > Changes made in this PR could be categorized in 3 different groups:
> > a) Needed changes because of changes in syntax or compiler being
> currently
> > more strict
> > b) Changes where a workaround is present but they are (or will be) fixed
> in
> > upcoming Scala versions
> > c) Changes in the build file
> >
> > In order to have a smooth and manageable migration to Scala 3, I propose
> > perform the migration in several steps:
> >
> > - Firstly apply all changes that would fall into category *a)*
> > - Once a new Scala version that incorporates the mentioned fixes is
> > released, revisit point changes in point *b)*
> > - Once Type erased for by-name parameters lampepfl/dotty#13638
> > <https://github.com/lampepfl/dotty/issues/13638> is resolved or a
> > workaround is found and Gradle with Scala 3 support is released we
> could
> > tackle the final step and incorporate changes mentioned in point *c)*
> > - After this, we would run the Scala Migration Tool to automatically
> > rewrite the code that uses removed syntax or language constructs. (See
> >
> >
> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
> > )
> >
> >
> > What do you think about this approach? Please feel free to share your
> > opinion on the proposed plan.
> > Also, I'm not sure if this should be written in a KIP form or not, no
> > public interfaces needed to be modified.
> >
> >
> > For extra information, these are the bugs reported to Scala 3 so far:
> >
> > - https://github.com/lampepfl/dotty/issues/13549
> > - https://github.com/lampepfl/dotty/issues/13572
> > - https://github.com/lampepfl/dotty/issues/13630
> > - https://github.com/lampepfl/dotty/issues/13638
> > - https://github.com/lampepfl/dotty/issues/13645
> >
> >
> > Thank you in advance!
> > --
> >
> > Josep Prat
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491715557497
> >
> > *w:* aiven.io
> >
> > *e:* josep.prat@aiven.io
> >
>
Re: [DISCUSS] Bringing Kafka towards Scala 3
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Josep,
Thanks for looking into this. As you said, it seems like there are a number
of rough edges still. Do we have many users asking for this right now? If
not, it may make sense to wait for things to get a bit more mature before
adding the burden of a third Scala version. Ideally, we would drop Scala
2.12 before we add Scala 3 support.
Ismael
On Fri, Oct 8, 2021 at 6:07 AM Josep Prat <jo...@aiven.io.invalid>
wrote:
> Hi there,
> For the last month I was actively working on migrating Apache Kafka to
> Scala 3. For what I could see, Apache Kafka is the biggest mixed Java-Scala
> project to attempt a migration to Scala 3. This means, a lot of regressions
> in regards to Java and Scala interoperability at bytecode were found within
> this task. You can take a look at the PR here:
> https://github.com/apache/kafka/pull/11350
> Currently, the status of the migration is as follows:
>
> - All tests pass in Scala 3 except one (
>
> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition).
> Root cause for this is this bug with no easy workaround: Type erased for
> by-name parameters lampepfl/dotty#13638
> <https://github.com/lampepfl/dotty/issues/13638>
> - All tests pass in Scala 2
> - It still uses a snapshot build of Gradle with support for Scala 3
> - Spotbugs detected some new errors when the code was compiled with
> Scala 3. Some are simply false positives, but a small subset seem like
> they
> might be legit bugs in Scala 3. I still need to come up with a minimized
> reproducer for those cases.
>
> As one can see, most of the changes are arguably low-level and no big
> rewrite was needed. The biggest refactoring was splitting several classes
> into their own file.
>
> Changes made in this PR could be categorized in 3 different groups:
> a) Needed changes because of changes in syntax or compiler being currently
> more strict
> b) Changes where a workaround is present but they are (or will be) fixed in
> upcoming Scala versions
> c) Changes in the build file
>
> In order to have a smooth and manageable migration to Scala 3, I propose
> perform the migration in several steps:
>
> - Firstly apply all changes that would fall into category *a)*
> - Once a new Scala version that incorporates the mentioned fixes is
> released, revisit point changes in point *b)*
> - Once Type erased for by-name parameters lampepfl/dotty#13638
> <https://github.com/lampepfl/dotty/issues/13638> is resolved or a
> workaround is found and Gradle with Scala 3 support is released we could
> tackle the final step and incorporate changes mentioned in point *c)*
> - After this, we would run the Scala Migration Tool to automatically
> rewrite the code that uses removed syntax or language constructs. (See
>
> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html
> )
>
>
> What do you think about this approach? Please feel free to share your
> opinion on the proposed plan.
> Also, I'm not sure if this should be written in a KIP form or not, no
> public interfaces needed to be modified.
>
>
> For extra information, these are the bugs reported to Scala 3 so far:
>
> - https://github.com/lampepfl/dotty/issues/13549
> - https://github.com/lampepfl/dotty/issues/13572
> - https://github.com/lampepfl/dotty/issues/13630
> - https://github.com/lampepfl/dotty/issues/13638
> - https://github.com/lampepfl/dotty/issues/13645
>
>
> Thank you in advance!
> --
>
> Josep Prat
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491715557497
>
> *w:* aiven.io
>
> *e:* josep.prat@aiven.io
>