You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Bryan Cutler <cu...@gmail.com> on 2019/11/18 05:16:08 UTC
[Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible
to prevent errors
After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above are
required to set the JVM property "io.netty.tryReflectionSetAccessible=true"
at startup, each time Arrow code is run, as documented at [2]. Not doing
this will result in the error "java.lang.UnsupportedOperationException:
sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available".
This is due to a part of the Netty codebase, and I'm not sure if there any
way around it, but I don't think it's the correct behavior for Arrow
out-of-the-box to fail with a 3rd party error by default. This has come up
before in our own unit testing [3], and most recently when upgrading Arrow
in Spark [4].
I'd like to propose that Arrow Java change to the following behavior:
1) check to see if the property io.netty.tryReflectionSetAccessible has
been set
2) if not set, automatically set to "true"
3) else if set to "false", catch the Netty error and prepend the error
message with the suggested setting of "true"
What do other devs think?
[1] https://issues.apache.org/jira/browse/ARROW-3191
[2] https://github.com/apache/arrow/tree/master/java#java-properties
[3] https://issues.apache.org/jira/browse/ARROW-5412
[4] https://github.com/apache/spark/pull/26552
Re: [Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible
to prevent errors
Posted by Bryan Cutler <cu...@gmail.com>.
I'm not sure what the best way to handle this is. Ideally we would use an
alternative that doesn't require setting a property, but I don't know Netty
well enough to make a recommendation. I also want to be careful not to
introduce anything that would hurt performance or cause any other side
effects. I made https://issues.apache.org/jira/browse/ARROW-7223 to track
this, we can continue the discussion there and I will try to do some
research into possible solutions.
On Wed, Nov 20, 2019 at 2:51 AM Fan Liya <li...@gmail.com> wrote:
> Hi Bryan,
>
> Thanks for bringing this up.
> +1 for the change.
>
> I am not clear what is the right place to override the jvm property.
> It is possible that when we try to override it (possibly in a static
> block), the old property value has already been read by netty library.
> To avoid this problem, do we need to control the order of class loading?
>
> Best,
> Liya Fan
>
> On Mon, Nov 18, 2019 at 3:17 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > This sounds reasonable to me. At this point I think having our consumers
> > have a better experience is more important then library purity concerns
> > I've had in the past.
> >
> > Do we need to handle jdk8 as a special case? Do you think it pays to try
> > to find an alternate library that doesn't require special flags for
> > whatever we are using this functionality for?
> >
> > Thanks,
> > Micah
> >
> > On Sunday, November 17, 2019, Bryan Cutler <cu...@gmail.com> wrote:
> >
> > > After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above
> are
> > > required to set the JVM property "io.netty.tryReflectionSetAccessible=
> > > true"
> > > at startup, each time Arrow code is run, as documented at [2]. Not
> doing
> > > this will result in the error "java.lang.UnsupportedOperationException:
> > > sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not
> available".
> > > This is due to a part of the Netty codebase, and I'm not sure if there
> > any
> > > way around it, but I don't think it's the correct behavior for Arrow
> > > out-of-the-box to fail with a 3rd party error by default. This has come
> > up
> > > before in our own unit testing [3], and most recently when upgrading
> > Arrow
> > > in Spark [4].
> > >
> > > I'd like to propose that Arrow Java change to the following behavior:
> > >
> > > 1) check to see if the property io.netty.tryReflectionSetAccessible has
> > > been set
> > > 2) if not set, automatically set to "true"
> > > 3) else if set to "false", catch the Netty error and prepend the error
> > > message with the suggested setting of "true"
> > >
> > > What do other devs think?
> > >
> > > [1] https://issues.apache.org/jira/browse/ARROW-3191
> > > [2] https://github.com/apache/arrow/tree/master/java#java-properties
> > > [3] https://issues.apache.org/jira/browse/ARROW-5412
> > > [4] https://github.com/apache/spark/pull/26552
> > >
> >
>
Re: [Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible
to prevent errors
Posted by Fan Liya <li...@gmail.com>.
Hi Bryan,
Thanks for bringing this up.
+1 for the change.
I am not clear what is the right place to override the jvm property.
It is possible that when we try to override it (possibly in a static
block), the old property value has already been read by netty library.
To avoid this problem, do we need to control the order of class loading?
Best,
Liya Fan
On Mon, Nov 18, 2019 at 3:17 PM Micah Kornfield <em...@gmail.com>
wrote:
> This sounds reasonable to me. At this point I think having our consumers
> have a better experience is more important then library purity concerns
> I've had in the past.
>
> Do we need to handle jdk8 as a special case? Do you think it pays to try
> to find an alternate library that doesn't require special flags for
> whatever we are using this functionality for?
>
> Thanks,
> Micah
>
> On Sunday, November 17, 2019, Bryan Cutler <cu...@gmail.com> wrote:
>
> > After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above are
> > required to set the JVM property "io.netty.tryReflectionSetAccessible=
> > true"
> > at startup, each time Arrow code is run, as documented at [2]. Not doing
> > this will result in the error "java.lang.UnsupportedOperationException:
> > sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available".
> > This is due to a part of the Netty codebase, and I'm not sure if there
> any
> > way around it, but I don't think it's the correct behavior for Arrow
> > out-of-the-box to fail with a 3rd party error by default. This has come
> up
> > before in our own unit testing [3], and most recently when upgrading
> Arrow
> > in Spark [4].
> >
> > I'd like to propose that Arrow Java change to the following behavior:
> >
> > 1) check to see if the property io.netty.tryReflectionSetAccessible has
> > been set
> > 2) if not set, automatically set to "true"
> > 3) else if set to "false", catch the Netty error and prepend the error
> > message with the suggested setting of "true"
> >
> > What do other devs think?
> >
> > [1] https://issues.apache.org/jira/browse/ARROW-3191
> > [2] https://github.com/apache/arrow/tree/master/java#java-properties
> > [3] https://issues.apache.org/jira/browse/ARROW-5412
> > [4] https://github.com/apache/spark/pull/26552
> >
>
Re: [Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible
to prevent errors
Posted by Micah Kornfield <em...@gmail.com>.
This sounds reasonable to me. At this point I think having our consumers
have a better experience is more important then library purity concerns
I've had in the past.
Do we need to handle jdk8 as a special case? Do you think it pays to try
to find an alternate library that doesn't require special flags for
whatever we are using this functionality for?
Thanks,
Micah
On Sunday, November 17, 2019, Bryan Cutler <cu...@gmail.com> wrote:
> After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above are
> required to set the JVM property "io.netty.tryReflectionSetAccessible=
> true"
> at startup, each time Arrow code is run, as documented at [2]. Not doing
> this will result in the error "java.lang.UnsupportedOperationException:
> sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available".
> This is due to a part of the Netty codebase, and I'm not sure if there any
> way around it, but I don't think it's the correct behavior for Arrow
> out-of-the-box to fail with a 3rd party error by default. This has come up
> before in our own unit testing [3], and most recently when upgrading Arrow
> in Spark [4].
>
> I'd like to propose that Arrow Java change to the following behavior:
>
> 1) check to see if the property io.netty.tryReflectionSetAccessible has
> been set
> 2) if not set, automatically set to "true"
> 3) else if set to "false", catch the Netty error and prepend the error
> message with the suggested setting of "true"
>
> What do other devs think?
>
> [1] https://issues.apache.org/jira/browse/ARROW-3191
> [2] https://github.com/apache/arrow/tree/master/java#java-properties
> [3] https://issues.apache.org/jira/browse/ARROW-5412
> [4] https://github.com/apache/spark/pull/26552
>