You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Niels Basjes <Ni...@basjes.nl> on 2023/05/05 10:44:43 UTC
[JAVA] Proposal: Dropping theUnsafe
Hi,
Currently several build plugins cannot be upgraded because the newer
versions require Java 11+.
So I'm working on this and I have a partially working pull request
https://issues.apache.org/jira/browse/AVRO-3716
https://github.com/apache/avro/pull/2118
One of the things I ran into is that currently the main library MUST be
built with JDK 8 because otherwise the code referring to theUnsafe simply
won't build.
Now there is already special code in place to use a reflection based system
that is used if you are not running on Java 8.
Since "everyone" I know is already running on Java 11 and 17 I propose to
- kick theUnsafe code, making the code in that area a lot simpler.
- Build the entire project with JDK 17 (or 11, but I prefer going to the
latest LTS version)
- Make it still produce Java 8 compliant code where possible (so just about
everywhere but not Thrift).
- Have checks in place to do a simple basic test using Java 8 (toolchains)
One of the effects is that the build on Github will no longer use the
"matrix" build and will build the project only once (I expect it to
become faster because of that).
Your opinions on this?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Christophe Le Saëc <ch...@gmail.com>.
For code replacing unsafe, may be i should better use VarHandle class like
explain in this oracle blog
<https://blogs.oracle.com/javamagazine/post/the-unsafe-class-unsafe-at-any-speed>
?
Le ven. 5 mai 2023 à 23:08, Fokko Driesprong <fo...@apache.org> a écrit :
> Hi Niels,
>
> Thanks for raising this. I'm having trouble understanding what you're
> running into. Looks like the CI pass on Java 11+
> <
> https://github.com/apache/avro/blob/master/.github/workflows/test-lang-java.yml#L41-L46
> >?
> Are the Unsafe paths not hit by the tests?
>
> If it helps, four years ago, I also did some similar changes in Parquet
> <https://github.com/apache/parquet-mr/pull/654>. I think we can port these
> changes also to Avro. Unfortunately, we still need to support Java8, since
> big
> projects like Hive <https://issues.apache.org/jira/browse/HIVE-22415> are
> still not on Java11. Related to this, I also opened up a PR to bring back
> Java 8 support for Thrift <https://github.com/apache/thrift/pull/2785>, I
> don't think that we can drop this soon.
>
> Kind regards,
> Fokko Driesprong
>
>
> Op vr 5 mei 2023 om 15:08 schreef Christophe Le Saëc <ch...@gmail.com>:
>
> > As far as i understood, MethodHandle is design to replace Unsafe call
> > <
> >
> http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html
> > >
> > If not, forget my first message.
> >
> > Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
> > oscar@westravanholthe.nl> a écrit :
> >
> > > Hi Niels,
> > >
> > > This is a very good plan. Though personally I prefer using the latest
> LTS
> > > version, I think we should require the oldest, actively supported LST
> > > version. For the next 5 months, this means Java 11.
> > >
> > > This link may be of interest here: https://endoflife.date/java
> > >
> > >
> > > Kind regards,
> > > Oscar
> > >
> > > On Fri, 5 May 2023 at 12:44, Niels Basjes <Ni...@basjes.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently several build plugins cannot be upgraded because the newer
> > > > versions require Java 11+.
> > > > So I'm working on this and I have a partially working pull request
> > > >
> > > > https://issues.apache.org/jira/browse/AVRO-3716
> > > > https://github.com/apache/avro/pull/2118
> > > >
> > > > One of the things I ran into is that currently the main library MUST
> be
> > > > built with JDK 8 because otherwise the code referring to theUnsafe
> > simply
> > > > won't build.
> > > > Now there is already special code in place to use a reflection based
> > > system
> > > > that is used if you are not running on Java 8.
> > > >
> > > > Since "everyone" I know is already running on Java 11 and 17 I
> propose
> > to
> > > > - kick theUnsafe code, making the code in that area a lot simpler.
> > > > - Build the entire project with JDK 17 (or 11, but I prefer going to
> > the
> > > > latest LTS version)
> > > > - Make it still produce Java 8 compliant code where possible (so just
> > > about
> > > > everywhere but not Thrift).
> > > > - Have checks in place to do a simple basic test using Java 8
> > > (toolchains)
> > > >
> > > > One of the effects is that the build on Github will no longer use the
> > > > "matrix" build and will build the project only once (I expect it to
> > > > become faster because of that).
> > > >
> > > > Your opinions on this?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> > >
> > > --
> > >
> > > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> > >
> >
>
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
Yes, the fact that the build passes on JDK 11 is a mystery to me at this
point.
I'll have a look at the links you mentioned.
I also really like your Java 8 effort towards Thrift. Thanks
Niels
On Fri, May 5, 2023 at 11:08 PM Fokko Driesprong <fo...@apache.org> wrote:
> Hi Niels,
>
> Thanks for raising this. I'm having trouble understanding what you're
> running into. Looks like the CI pass on Java 11+
> <
> https://github.com/apache/avro/blob/master/.github/workflows/test-lang-java.yml#L41-L46
> >?
> Are the Unsafe paths not hit by the tests?
>
> If it helps, four years ago, I also did some similar changes in Parquet
> <https://github.com/apache/parquet-mr/pull/654>. I think we can port these
> changes also to Avro. Unfortunately, we still need to support Java8, since
> big
> projects like Hive <https://issues.apache.org/jira/browse/HIVE-22415> are
> still not on Java11. Related to this, I also opened up a PR to bring back
> Java 8 support for Thrift <https://github.com/apache/thrift/pull/2785>, I
> don't think that we can drop this soon.
>
> Kind regards,
> Fokko Driesprong
>
>
> Op vr 5 mei 2023 om 15:08 schreef Christophe Le Saëc <ch...@gmail.com>:
>
> > As far as i understood, MethodHandle is design to replace Unsafe call
> > <
> >
> http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html
> > >
> > If not, forget my first message.
> >
> > Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
> > oscar@westravanholthe.nl> a écrit :
> >
> > > Hi Niels,
> > >
> > > This is a very good plan. Though personally I prefer using the latest
> LTS
> > > version, I think we should require the oldest, actively supported LST
> > > version. For the next 5 months, this means Java 11.
> > >
> > > This link may be of interest here: https://endoflife.date/java
> > >
> > >
> > > Kind regards,
> > > Oscar
> > >
> > > On Fri, 5 May 2023 at 12:44, Niels Basjes <Ni...@basjes.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently several build plugins cannot be upgraded because the newer
> > > > versions require Java 11+.
> > > > So I'm working on this and I have a partially working pull request
> > > >
> > > > https://issues.apache.org/jira/browse/AVRO-3716
> > > > https://github.com/apache/avro/pull/2118
> > > >
> > > > One of the things I ran into is that currently the main library MUST
> be
> > > > built with JDK 8 because otherwise the code referring to theUnsafe
> > simply
> > > > won't build.
> > > > Now there is already special code in place to use a reflection based
> > > system
> > > > that is used if you are not running on Java 8.
> > > >
> > > > Since "everyone" I know is already running on Java 11 and 17 I
> propose
> > to
> > > > - kick theUnsafe code, making the code in that area a lot simpler.
> > > > - Build the entire project with JDK 17 (or 11, but I prefer going to
> > the
> > > > latest LTS version)
> > > > - Make it still produce Java 8 compliant code where possible (so just
> > > about
> > > > everywhere but not Thrift).
> > > > - Have checks in place to do a simple basic test using Java 8
> > > (toolchains)
> > > >
> > > > One of the effects is that the build on Github will no longer use the
> > > > "matrix" build and will build the project only once (I expect it to
> > > > become faster because of that).
> > > >
> > > > Your opinions on this?
> > > >
> > > > --
> > > > Best regards / Met vriendelijke groeten,
> > > >
> > > > Niels Basjes
> > > >
> > >
> > >
> > > --
> > >
> > > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> > >
> >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Fokko Driesprong <fo...@apache.org>.
Hi Niels,
Thanks for raising this. I'm having trouble understanding what you're
running into. Looks like the CI pass on Java 11+
<https://github.com/apache/avro/blob/master/.github/workflows/test-lang-java.yml#L41-L46>?
Are the Unsafe paths not hit by the tests?
If it helps, four years ago, I also did some similar changes in Parquet
<https://github.com/apache/parquet-mr/pull/654>. I think we can port these
changes also to Avro. Unfortunately, we still need to support Java8, since big
projects like Hive <https://issues.apache.org/jira/browse/HIVE-22415> are
still not on Java11. Related to this, I also opened up a PR to bring back
Java 8 support for Thrift <https://github.com/apache/thrift/pull/2785>, I
don't think that we can drop this soon.
Kind regards,
Fokko Driesprong
Op vr 5 mei 2023 om 15:08 schreef Christophe Le Saëc <ch...@gmail.com>:
> As far as i understood, MethodHandle is design to replace Unsafe call
> <
> http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html
> >
> If not, forget my first message.
>
> Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
> oscar@westravanholthe.nl> a écrit :
>
> > Hi Niels,
> >
> > This is a very good plan. Though personally I prefer using the latest LTS
> > version, I think we should require the oldest, actively supported LST
> > version. For the next 5 months, this means Java 11.
> >
> > This link may be of interest here: https://endoflife.date/java
> >
> >
> > Kind regards,
> > Oscar
> >
> > On Fri, 5 May 2023 at 12:44, Niels Basjes <Ni...@basjes.nl> wrote:
> >
> > > Hi,
> > >
> > > Currently several build plugins cannot be upgraded because the newer
> > > versions require Java 11+.
> > > So I'm working on this and I have a partially working pull request
> > >
> > > https://issues.apache.org/jira/browse/AVRO-3716
> > > https://github.com/apache/avro/pull/2118
> > >
> > > One of the things I ran into is that currently the main library MUST be
> > > built with JDK 8 because otherwise the code referring to theUnsafe
> simply
> > > won't build.
> > > Now there is already special code in place to use a reflection based
> > system
> > > that is used if you are not running on Java 8.
> > >
> > > Since "everyone" I know is already running on Java 11 and 17 I propose
> to
> > > - kick theUnsafe code, making the code in that area a lot simpler.
> > > - Build the entire project with JDK 17 (or 11, but I prefer going to
> the
> > > latest LTS version)
> > > - Make it still produce Java 8 compliant code where possible (so just
> > about
> > > everywhere but not Thrift).
> > > - Have checks in place to do a simple basic test using Java 8
> > (toolchains)
> > >
> > > One of the effects is that the build on Github will no longer use the
> > > "matrix" build and will build the project only once (I expect it to
> > > become faster because of that).
> > >
> > > Your opinions on this?
> > >
> > > --
> > > Best regards / Met vriendelijke groeten,
> > >
> > > Niels Basjes
> > >
> >
> >
> > --
> >
> > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> >
>
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Christophe Le Saëc <ch...@gmail.com>.
As far as i understood, MethodHandle is design to replace Unsafe call
<http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html>
If not, forget my first message.
Le ven. 5 mai 2023 à 14:37, Oscar Westra van Holthe - Kind <
oscar@westravanholthe.nl> a écrit :
> Hi Niels,
>
> This is a very good plan. Though personally I prefer using the latest LTS
> version, I think we should require the oldest, actively supported LST
> version. For the next 5 months, this means Java 11.
>
> This link may be of interest here: https://endoflife.date/java
>
>
> Kind regards,
> Oscar
>
> On Fri, 5 May 2023 at 12:44, Niels Basjes <Ni...@basjes.nl> wrote:
>
> > Hi,
> >
> > Currently several build plugins cannot be upgraded because the newer
> > versions require Java 11+.
> > So I'm working on this and I have a partially working pull request
> >
> > https://issues.apache.org/jira/browse/AVRO-3716
> > https://github.com/apache/avro/pull/2118
> >
> > One of the things I ran into is that currently the main library MUST be
> > built with JDK 8 because otherwise the code referring to theUnsafe simply
> > won't build.
> > Now there is already special code in place to use a reflection based
> system
> > that is used if you are not running on Java 8.
> >
> > Since "everyone" I know is already running on Java 11 and 17 I propose to
> > - kick theUnsafe code, making the code in that area a lot simpler.
> > - Build the entire project with JDK 17 (or 11, but I prefer going to the
> > latest LTS version)
> > - Make it still produce Java 8 compliant code where possible (so just
> about
> > everywhere but not Thrift).
> > - Have checks in place to do a simple basic test using Java 8
> (toolchains)
> >
> > One of the effects is that the build on Github will no longer use the
> > "matrix" build and will build the project only once (I expect it to
> > become faster because of that).
> >
> > Your opinions on this?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
> --
>
> ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
>
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi Niels,
This is a very good plan. Though personally I prefer using the latest LTS
version, I think we should require the oldest, actively supported LST
version. For the next 5 months, this means Java 11.
This link may be of interest here: https://endoflife.date/java
Kind regards,
Oscar
On Fri, 5 May 2023 at 12:44, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> Currently several build plugins cannot be upgraded because the newer
> versions require Java 11+.
> So I'm working on this and I have a partially working pull request
>
> https://issues.apache.org/jira/browse/AVRO-3716
> https://github.com/apache/avro/pull/2118
>
> One of the things I ran into is that currently the main library MUST be
> built with JDK 8 because otherwise the code referring to theUnsafe simply
> won't build.
> Now there is already special code in place to use a reflection based system
> that is used if you are not running on Java 8.
>
> Since "everyone" I know is already running on Java 11 and 17 I propose to
> - kick theUnsafe code, making the code in that area a lot simpler.
> - Build the entire project with JDK 17 (or 11, but I prefer going to the
> latest LTS version)
> - Make it still produce Java 8 compliant code where possible (so just about
> everywhere but not Thrift).
> - Have checks in place to do a simple basic test using Java 8 (toolchains)
>
> One of the effects is that the build on Github will no longer use the
> "matrix" build and will build the project only once (I expect it to
> become faster because of that).
>
> Your opinions on this?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
Curious: what is the advantage of the FieldAccessHandle you wrote over the
(now present) FieldAccessReflect?
i.e. : Why did you create this alternative?
Niels
On Fri, May 5, 2023 at 1:47 PM Christophe Le Saëc <ch...@gmail.com>
wrote:
> Hello Niels,
> This is a nice idea !!
> In same spirit, few time ago, i wrote class FieldAccessHandle (see joined
> file), another extension of FieldAccess, that use
> "java.lang.invoke.MethodHandle" instead of Unsafe.
> And change ReflectionUtil class with it
>
> final FieldAccess unsafeAccess;
> if (javaVersion != null && javaVersion.startsWith("1.8")) {
> unsafeAccess = load("org.apache.avro.reflect.FieldAccessUnsafe",
> FieldAccess.class);
> } else {
> unsafeAccess = load("org.apache.avro.reflect.FieldAccessHandle",
> FieldAccess.class);
>
> }
>
> Hope it could help
>
> Le ven. 5 mai 2023 à 12:45, Niels Basjes <Ni...@basjes.nl> a écrit :
>
>> Hi,
>>
>> Currently several build plugins cannot be upgraded because the newer
>> versions require Java 11+.
>> So I'm working on this and I have a partially working pull request
>>
>> https://issues.apache.org/jira/browse/AVRO-3716
>> https://github.com/apache/avro/pull/2118
>>
>> One of the things I ran into is that currently the main library MUST be
>> built with JDK 8 because otherwise the code referring to theUnsafe simply
>> won't build.
>> Now there is already special code in place to use a reflection based
>> system
>> that is used if you are not running on Java 8.
>>
>> Since "everyone" I know is already running on Java 11 and 17 I propose to
>> - kick theUnsafe code, making the code in that area a lot simpler.
>> - Build the entire project with JDK 17 (or 11, but I prefer going to the
>> latest LTS version)
>> - Make it still produce Java 8 compliant code where possible (so just
>> about
>> everywhere but not Thrift).
>> - Have checks in place to do a simple basic test using Java 8 (toolchains)
>>
>> One of the effects is that the build on Github will no longer use the
>> "matrix" build and will build the project only once (I expect it to
>> become faster because of that).
>>
>> Your opinions on this?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [JAVA] Proposal: Dropping theUnsafe
Posted by Christophe Le Saëc <ch...@gmail.com>.
Hello Niels,
This is a nice idea !!
In same spirit, few time ago, i wrote class FieldAccessHandle (see joined
file), another extension of FieldAccess, that use
"java.lang.invoke.MethodHandle" instead of Unsafe.
And change ReflectionUtil class with it
final FieldAccess unsafeAccess;
if (javaVersion != null && javaVersion.startsWith("1.8")) {
unsafeAccess = load("org.apache.avro.reflect.FieldAccessUnsafe",
FieldAccess.class);
} else {
unsafeAccess = load("org.apache.avro.reflect.FieldAccessHandle",
FieldAccess.class);
}
Hope it could help
Le ven. 5 mai 2023 à 12:45, Niels Basjes <Ni...@basjes.nl> a écrit :
> Hi,
>
> Currently several build plugins cannot be upgraded because the newer
> versions require Java 11+.
> So I'm working on this and I have a partially working pull request
>
> https://issues.apache.org/jira/browse/AVRO-3716
> https://github.com/apache/avro/pull/2118
>
> One of the things I ran into is that currently the main library MUST be
> built with JDK 8 because otherwise the code referring to theUnsafe simply
> won't build.
> Now there is already special code in place to use a reflection based system
> that is used if you are not running on Java 8.
>
> Since "everyone" I know is already running on Java 11 and 17 I propose to
> - kick theUnsafe code, making the code in that area a lot simpler.
> - Build the entire project with JDK 17 (or 11, but I prefer going to the
> latest LTS version)
> - Make it still produce Java 8 compliant code where possible (so just about
> everywhere but not Thrift).
> - Have checks in place to do a simple basic test using Java 8 (toolchains)
>
> One of the effects is that the build on Github will no longer use the
> "matrix" build and will build the project only once (I expect it to
> become faster because of that).
>
> Your opinions on this?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>