You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Niels Basjes <Ni...@basjes.nl> on 2023/03/08 22:22:04 UTC
[DISCUSS] Java 8 compatibility?
Hi,
I was looking into making the build run in only Java 11 or 17 because there
are more and more maven plugins that only run in those.
Newer versions of for example the maven-checkstyle-plugin need Java 11.
https://github.com/apache/avro/pull/2118
I consider that to be a low risk change because we can make sure that the
code still runs under java 8 to support all of the older systems that use
Avro.
Having some of the integration tests run via invoker and then use
toolchains is a perfect way to make something like that work.
Just now I noticed that key dependencies (not plugins: actual dependencies)
have started dropping Java 8 support.
The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
0.18.1 in /lang/java https://github.com/apache/avro/pull/2124 ) fails with:
*[INFO] Restricted to JDK 8 yet
org.apache.thrift:libthrift:jar:0.18.1:compile contains
org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
*I have asked the guys at Thrift if they can have the next release be Java
8 again, let's see if they can.*
Avro is a dependency for many projects of which a significant group (small
projects like Hadoop and Hive) still need Java 8 and cannot build under
Java 11.
So we cannot simply switch to Java 11.
One thing I thought of is to switch "partially".
Something like all modules be Java 8 compatible, except the Thrift one
which is Java 11.
Like to hear your thoughts on this.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [DISCUSS] Java 8 compatibility?
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi all,
The extra effort required to support aan LTS version, as opposed to
'merely' refraining from using new language features in one or 2 modules,
is a poor use of our volunteered time IMHO.
The advantage of Niels suggestion is also that projects benefitting a Java
>8 Avro module already use dependencies that require that higher Java
version. It is a very user friendly way to (partially) drop Java 8 support.
Another question: if we target Java >8 for a module, should that use the
Java Module system as well?
Kind regards,
Oscar
--
Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
Op vr 10 mrt. 2023 15:52 schreef Niels Basjes <Ni...@basjes.nl>:
> Hi,
>
> Because of the harder maintenance I usually do not favor having an LTS
> to manage as well.
>
> I like your last sentence more.
>
> Let me summarize how I see this:
>
> 1. The core libraries (i.e. core, compiler,... ) remain Java 8 because
> that is all that is needed... I think.
> 2. For each context that is specific for something else (like thrift,
> trevni, mapred, etc.) the java version is highest of java 8 and what
> that
> context needs.
> 1. So for thrift this means Java 11. If a downstream dependency
> cannot handle Java 11 then they are stuck and cannot upgrade Avro
> anymore
> because they should but cannot upgrade thrift.
> 2. We can still build our code into java 8 classes, but because of
> the required context that won't help anyone.
>
> Consequences I see now:
>
> - We only have 1 release.
> - There is no LTS to maintain separately which makes it all easier.
> - All dependencies are always up to date.
> - If you need the component that works with X then you get the
> integration for the latest version of X.
> - If someone finds a security problem in "the last jdk8 version"
> of a dependency that will no longer get any updates: we don't
> have any
> problems.
> - Alternatively if we would have an LTS then we WILL have problems
> because there will not be an update available.
> - The build has different JDK settings depending on the module.
> - We can run the entire build under Java 17 (i.e. only once)
> - Which should make this part of the build a lot faster.
> - Toolchains can be used to control the exact JDK needed for the
> build.
> - We can use integration tests with toolchains and something like
> this
>
> https://www.mojohaus.org/extra-enforcer-rules/enforceBytecodeVersion.html
> to make sure it all works with the correct Java version.
>
>
> Note: I use this model also in this project of mine where I have a core and
> a lot of UDFs in which this core is wrapped.
> https://github.com/nielsbasjes/yauaa
>
>
>
> On Thu, Mar 9, 2023 at 10:07 PM Ryan Skraba <ry...@skraba.com> wrote:
>
> > This seems like something that an LTS version might serve well! For
> > as long as I've used Avro, only the most recent version has active
> > support, but we've talked about having more than one version.
> >
> > If we go this route, I'd be comfortable dropping Java 8 on Avro
> > 1.12.x! I'd also be comfortable only supporting core avro with only
> > Java 8.
> >
> > All my best, Ryan
> >
> > On Thu, Mar 9, 2023 at 10:28 AM Christophe Le Saëc <ch...@gmail.com>
> > wrote:
> > >
> > > Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11
> > runtime
> > > support is completed."
> > > For Hive, it seems to be a question of time
> > > <https://issues.apache.org/jira/browse/HIVE-22415> 😉, to remove
> > joda.time
> > > lib
> > >
> > > More generally, could we consider that projects which need Java8 or old
> > > library have to use old avro version (like beam uses Avro 1.8.2 version
> > > <
> >
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527
> > >
> > > ).
> > >
> > > So, if we maintain Java8 compatibility for current avro versions, make
> > the
> > > future "major" version in Java 11 or 17 only is not a pb i think.
> > >
> > > Best regards,
> > > Christophe Le Saëc
> > >
> > >
> > > Le mer. 8 mars 2023 à 23:22, Niels Basjes <Ni...@basjes.nl> a écrit :
> > >
> > > > Hi,
> > > >
> > > > I was looking into making the build run in only Java 11 or 17 because
> > there
> > > > are more and more maven plugins that only run in those.
> > > > Newer versions of for example the maven-checkstyle-plugin need Java
> 11.
> > > > https://github.com/apache/avro/pull/2118
> > > >
> > > > I consider that to be a low risk change because we can make sure that
> > the
> > > > code still runs under java 8 to support all of the older systems that
> > use
> > > > Avro.
> > > > Having some of the integration tests run via invoker and then use
> > > > toolchains is a perfect way to make something like that work.
> > > >
> > > > Just now I noticed that key dependencies (not plugins: actual
> > dependencies)
> > > > have started dropping Java 8 support.
> > > >
> > > > The update to the latest Thrift library ( Bump libthrift from 0.16.0
> to
> > > > 0.18.1 in /lang/java https://github.com/apache/avro/pull/2124 )
> fails
> > > > with:
> > > > *[INFO] Restricted to JDK 8 yet
> > > > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > > > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> > > >
> > > >
> > > > *I have asked the guys at Thrift if they can have the next release be
> > Java
> > > > 8 again, let's see if they can.*
> > > >
> > > > Avro is a dependency for many projects of which a significant group
> > (small
> > > > projects like Hadoop and Hive) still need Java 8 and cannot build
> under
> > > > Java 11.
> > > > So we cannot simply switch to Java 11.
> > > >
> > > > One thing I thought of is to switch "partially".
> > > > Something like all modules be Java 8 compatible, except the Thrift
> one
> > > > which is Java 11.
> > > >
> > > > Like to hear your thoughts on this.
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> >
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Re: [DISCUSS] Java 8 compatibility?
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
Because of the harder maintenance I usually do not favor having an LTS
to manage as well.
I like your last sentence more.
Let me summarize how I see this:
1. The core libraries (i.e. core, compiler,... ) remain Java 8 because
that is all that is needed... I think.
2. For each context that is specific for something else (like thrift,
trevni, mapred, etc.) the java version is highest of java 8 and what that
context needs.
1. So for thrift this means Java 11. If a downstream dependency
cannot handle Java 11 then they are stuck and cannot upgrade Avro anymore
because they should but cannot upgrade thrift.
2. We can still build our code into java 8 classes, but because of
the required context that won't help anyone.
Consequences I see now:
- We only have 1 release.
- There is no LTS to maintain separately which makes it all easier.
- All dependencies are always up to date.
- If you need the component that works with X then you get the
integration for the latest version of X.
- If someone finds a security problem in "the last jdk8 version"
of a dependency that will no longer get any updates: we don't have any
problems.
- Alternatively if we would have an LTS then we WILL have problems
because there will not be an update available.
- The build has different JDK settings depending on the module.
- We can run the entire build under Java 17 (i.e. only once)
- Which should make this part of the build a lot faster.
- Toolchains can be used to control the exact JDK needed for the
build.
- We can use integration tests with toolchains and something like
this
https://www.mojohaus.org/extra-enforcer-rules/enforceBytecodeVersion.html
to make sure it all works with the correct Java version.
Note: I use this model also in this project of mine where I have a core and
a lot of UDFs in which this core is wrapped.
https://github.com/nielsbasjes/yauaa
On Thu, Mar 9, 2023 at 10:07 PM Ryan Skraba <ry...@skraba.com> wrote:
> This seems like something that an LTS version might serve well! For
> as long as I've used Avro, only the most recent version has active
> support, but we've talked about having more than one version.
>
> If we go this route, I'd be comfortable dropping Java 8 on Avro
> 1.12.x! I'd also be comfortable only supporting core avro with only
> Java 8.
>
> All my best, Ryan
>
> On Thu, Mar 9, 2023 at 10:28 AM Christophe Le Saëc <ch...@gmail.com>
> wrote:
> >
> > Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11
> runtime
> > support is completed."
> > For Hive, it seems to be a question of time
> > <https://issues.apache.org/jira/browse/HIVE-22415> 😉, to remove
> joda.time
> > lib
> >
> > More generally, could we consider that projects which need Java8 or old
> > library have to use old avro version (like beam uses Avro 1.8.2 version
> > <
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527
> >
> > ).
> >
> > So, if we maintain Java8 compatibility for current avro versions, make
> the
> > future "major" version in Java 11 or 17 only is not a pb i think.
> >
> > Best regards,
> > Christophe Le Saëc
> >
> >
> > Le mer. 8 mars 2023 à 23:22, Niels Basjes <Ni...@basjes.nl> a écrit :
> >
> > > Hi,
> > >
> > > I was looking into making the build run in only Java 11 or 17 because
> there
> > > are more and more maven plugins that only run in those.
> > > Newer versions of for example the maven-checkstyle-plugin need Java 11.
> > > https://github.com/apache/avro/pull/2118
> > >
> > > I consider that to be a low risk change because we can make sure that
> the
> > > code still runs under java 8 to support all of the older systems that
> use
> > > Avro.
> > > Having some of the integration tests run via invoker and then use
> > > toolchains is a perfect way to make something like that work.
> > >
> > > Just now I noticed that key dependencies (not plugins: actual
> dependencies)
> > > have started dropping Java 8 support.
> > >
> > > The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
> > > 0.18.1 in /lang/java https://github.com/apache/avro/pull/2124 ) fails
> > > with:
> > > *[INFO] Restricted to JDK 8 yet
> > > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> > >
> > >
> > > *I have asked the guys at Thrift if they can have the next release be
> Java
> > > 8 again, let's see if they can.*
> > >
> > > Avro is a dependency for many projects of which a significant group
> (small
> > > projects like Hadoop and Hive) still need Java 8 and cannot build under
> > > Java 11.
> > > So we cannot simply switch to Java 11.
> > >
> > > One thing I thought of is to switch "partially".
> > > Something like all modules be Java 8 compatible, except the Thrift one
> > > which is Java 11.
> > >
> > > Like to hear your thoughts on this.
> > >
> > > --
> > > Best regards / Met vriendelijke groeten,
> > >
> > > Niels Basjes
> > >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [DISCUSS] Java 8 compatibility?
Posted by Ryan Skraba <ry...@skraba.com>.
This seems like something that an LTS version might serve well! For
as long as I've used Avro, only the most recent version has active
support, but we've talked about having more than one version.
If we go this route, I'd be comfortable dropping Java 8 on Avro
1.12.x! I'd also be comfortable only supporting core avro with only
Java 8.
All my best, Ryan
On Thu, Mar 9, 2023 at 10:28 AM Christophe Le Saëc <ch...@gmail.com> wrote:
>
> Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11 runtime
> support is completed."
> For Hive, it seems to be a question of time
> <https://issues.apache.org/jira/browse/HIVE-22415> 😉, to remove joda.time
> lib
>
> More generally, could we consider that projects which need Java8 or old
> library have to use old avro version (like beam uses Avro 1.8.2 version
> <https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527>
> ).
>
> So, if we maintain Java8 compatibility for current avro versions, make the
> future "major" version in Java 11 or 17 only is not a pb i think.
>
> Best regards,
> Christophe Le Saëc
>
>
> Le mer. 8 mars 2023 à 23:22, Niels Basjes <Ni...@basjes.nl> a écrit :
>
> > Hi,
> >
> > I was looking into making the build run in only Java 11 or 17 because there
> > are more and more maven plugins that only run in those.
> > Newer versions of for example the maven-checkstyle-plugin need Java 11.
> > https://github.com/apache/avro/pull/2118
> >
> > I consider that to be a low risk change because we can make sure that the
> > code still runs under java 8 to support all of the older systems that use
> > Avro.
> > Having some of the integration tests run via invoker and then use
> > toolchains is a perfect way to make something like that work.
> >
> > Just now I noticed that key dependencies (not plugins: actual dependencies)
> > have started dropping Java 8 support.
> >
> > The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
> > 0.18.1 in /lang/java https://github.com/apache/avro/pull/2124 ) fails
> > with:
> > *[INFO] Restricted to JDK 8 yet
> > org.apache.thrift:libthrift:jar:0.18.1:compile contains
> > org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
> >
> >
> > *I have asked the guys at Thrift if they can have the next release be Java
> > 8 again, let's see if they can.*
> >
> > Avro is a dependency for many projects of which a significant group (small
> > projects like Hadoop and Hive) still need Java 8 and cannot build under
> > Java 11.
> > So we cannot simply switch to Java 11.
> >
> > One thing I thought of is to switch "partially".
> > Something like all modules be Java 8 compatible, except the Thrift one
> > which is Java 11.
> >
> > Like to hear your thoughts on this.
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
Re: [DISCUSS] Java 8 compatibility?
Posted by Christophe Le Saëc <ch...@gmail.com>.
Hadoop doc says <https://hadoop.apache.org/docs/stable/> "Java 11 runtime
support is completed."
For Hive, it seems to be a question of time
<https://issues.apache.org/jira/browse/HIVE-22415> 😉, to remove joda.time
lib
More generally, could we consider that projects which need Java8 or old
library have to use old avro version (like beam uses Avro 1.8.2 version
<https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L527>
).
So, if we maintain Java8 compatibility for current avro versions, make the
future "major" version in Java 11 or 17 only is not a pb i think.
Best regards,
Christophe Le Saëc
Le mer. 8 mars 2023 à 23:22, Niels Basjes <Ni...@basjes.nl> a écrit :
> Hi,
>
> I was looking into making the build run in only Java 11 or 17 because there
> are more and more maven plugins that only run in those.
> Newer versions of for example the maven-checkstyle-plugin need Java 11.
> https://github.com/apache/avro/pull/2118
>
> I consider that to be a low risk change because we can make sure that the
> code still runs under java 8 to support all of the older systems that use
> Avro.
> Having some of the integration tests run via invoker and then use
> toolchains is a perfect way to make something like that work.
>
> Just now I noticed that key dependencies (not plugins: actual dependencies)
> have started dropping Java 8 support.
>
> The update to the latest Thrift library ( Bump libthrift from 0.16.0 to
> 0.18.1 in /lang/java https://github.com/apache/avro/pull/2124 ) fails
> with:
> *[INFO] Restricted to JDK 8 yet
> org.apache.thrift:libthrift:jar:0.18.1:compile contains
> org/apache/thrift/TBaseProcessor.class targeted to JDK 11*
>
>
> *I have asked the guys at Thrift if they can have the next release be Java
> 8 again, let's see if they can.*
>
> Avro is a dependency for many projects of which a significant group (small
> projects like Hadoop and Hive) still need Java 8 and cannot build under
> Java 11.
> So we cannot simply switch to Java 11.
>
> One thing I thought of is to switch "partially".
> Something like all modules be Java 8 compatible, except the Thrift one
> which is Java 11.
>
> Like to hear your thoughts on this.
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>