You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Rory O'Donnell <ro...@oracle.com> on 2016/12/09 10:33:50 UTC
JDK 9 b148 including a refresh of the module system is available on
java.net
Hi Uwe & Dawid,
JDK 9 build b148 <https://jdk9.java.net/download/> includes an important
Refresh of the module system [1] , summary of changes are listed here
<http://download.java.net/java/jdk9/changes/jdk-9+148.html>.
*This refresh includes a disruptive change that is important to understand.
*For those that have been trying out modules with regular JDK 9 builds
then be aware that `requires public` changes to `requires transitive`.
In addition, the binary representation of the module declaration
(module-info.class) has changed so that you need to recompile any
modules that were compiled with previous JDK 9 builds.
As things stand today in JDK 9 then you use setAccessible to break into
non-public elements of any type in exported packages. However, it cannot
be used to break into any type in non-exported package. The current
specified behavior was a compromise for the initial integration of the
module system. It is of course not very satisfactory, hence the
#AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With
the updated proposal in the JSR, this refresh changes setAccessible
further so that it cannot be used to break into non-public types, or
non-public elements of public types, in exported packages. Code that
uses setAccessible to hack into the private constructor of
java.lang.invoke.MethodHandles.Lookup will be disappointed for example.
This change will expose hacks in many existing libraries and tools. As a
workaround then a new command line option `--add-opens` can be used to
open specific packages for "deep reflection". For example, a really
popular build tool fails with this refresh because it uses setAccessible
+ core reflection to hack into a private field of an unmodifiable
collection so that it can mutate it, facepalm! This code will continue
to work as before when run with `--add-opens
java.base/java.util=ALL-UNNAMED` to open the package java.util in module
java.base to "all unnamed modules" (think class path).
*Any help reporting issues to popular tools and libraries would be
appreciated. *
A debugging aid that is useful to identify issues is to run with
-Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when
setAccessible fails, this is particularly useful when code swallows
exceptions without any logging.
Rgds,Rory
[1]
http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html
<http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html>
[2]
http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland
RE: JDK 9 b148 including a refresh of the module system is available on java.net
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Sorry my message was incorrect:
There are 2 bugs in Lucene and Java:
- In Lucene the unmap hack’s exception handling falls over the stupid InaccessibleObjectException does not extend ReflectiveOperationException issue (I complained about that several times). I can fix this, but it’s horrible.
- Java now hides internal of ByteBuffer completely, although the JEP about Unsafe & Co. said which Java APIs are excluded from the lock down. Next to Unsafe, also ByteBuffer was on the list of “still allowed APIs where the Java 9 backwards layer helps”. MappedByteBuffer unmapping is one of the most important parts in Lucene that must be supported, otherwise it is impossible to use Lucene, Solr, or Elasticsearch with Java 9 in a performant way. We can also not require from our users to add strange command line options to their JVMs.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Friday, December 9, 2016 5:53 PM
To: dev@lucene.apache.org; 'Rory O'Donnell' <ro...@oracle.com>
Cc: 'Dalibor Topic' <da...@oracle.com>; balchandra.vaidya@oracle.com; abdul.kolarkunnu@oracle.com; 'Dawid Weiss' <da...@cs.put.poznan.pl>
Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net
Hi,
build 148 broke the unmap hack. The problem is an additional bug in Java: The unmap hack should be more unspecific on the Exception Type (InaccessibleObjectException does not extend ReflectiveOperationException). I will change this in an issue, but the general problem persists:
With the buil 148 version we can no longer unmap byte buffers:
java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio" to unnamed module @10b73cf
[junit4] > at java.base/jdk.internal.reflect.Reflection.throwInaccessibleObjectException(Reflection.java:427)
[junit4] > at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:201)
[junit4] > at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:192)
[junit4] > at java.base/java.lang.reflect.Method.setAccessible(Method.java:186)
[junit4] > at org.apache.lucene.store.MMapDirectory.unmapHackImpl(MMapDirectory.java:345)
[junit4] > at java.base/java.security.AccessController.doPrivileged(Native Method)
[junit4] > at org.apache.lucene.store.MMapDirectory.<clinit>(MMapDirectory.java:326)
[junit4] > ... 42 moreThrowable #2: java.lang.NullPointerException
[junit4] > at org.apache.lucene.search.TestDateSort.tearDown(TestDateSort.java:71)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:538)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de <http://www.thetaphi.de/>
eMail: <ma...@thetaphi.de> uwe@thetaphi.de
From: Uwe Schindler [ <ma...@thetaphi.de> mailto:uwe@thetaphi.de]
Sent: Friday, December 9, 2016 4:35 PM
To: 'Rory O'Donnell' < <ma...@oracle.com> rory.odonnell@oracle.com>
Cc: 'Dalibor Topic' < <ma...@oracle.com> dalibor.topic@oracle.com>; <ma...@oracle.com> balchandra.vaidya@oracle.com; <ma...@oracle.com> abdul.kolarkunnu@oracle.com; 'Dawid Weiss' < <ma...@cs.put.poznan.pl> dawid.weiss@cs.put.poznan.pl>; <ma...@lucene.apache.org> dev@lucene.apache.org
Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net
Hi Rory,
thanks for the message! I put build 148 in the Jenkins test rotation. Let’s see what happens.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de <http://www.thetaphi.de/>
eMail: <ma...@thetaphi.de> uwe@thetaphi.de
From: Rory O'Donnell [ <ma...@oracle.com> mailto:rory.odonnell@oracle.com]
Sent: Friday, December 9, 2016 11:34 AM
To: <ma...@thetaphi.de> uwe@thetaphi.de
Cc: <ma...@oracle.com> rory.odonnell@oracle.com; Dalibor Topic < <ma...@oracle.com> dalibor.topic@oracle.com>; <ma...@oracle.com> balchandra.vaidya@oracle.com; <ma...@oracle.com> abdul.kolarkunnu@oracle.com; Dawid Weiss < <ma...@cs.put.poznan.pl> dawid.weiss@cs.put.poznan.pl>; <ma...@lucene.apache.org> dev@lucene.apache.org
Subject: JDK 9 b148 including a refresh of the module system is available on java.net
Hi Uwe & Dawid,
JDK 9 build b148 <https://jdk9.java.net/download/> includes an important Refresh of the module system [1] , summary of changes are listed here <http://download.java.net/java/jdk9/changes/jdk-9+148.html> .
This refresh includes a disruptive change that is important to understand.
For those that have been trying out modules with regular JDK 9 builds then be aware that `requires public` changes to `requires transitive`. In addition, the binary representation of the module declaration (module-info.class) has changed so that you need to recompile any modules that were compiled with previous JDK 9 builds.
As things stand today in JDK 9 then you use setAccessible to break into non-public elements of any type in exported packages. However, it cannot be used to break into any type in non-exported package. The current specified behavior was a compromise for the initial integration of the module system. It is of course not very satisfactory, hence the #AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With the updated proposal in the JSR, this refresh changes setAccessible further so that it cannot be used to break into non-public types, or non-public elements of public types, in exported packages. Code that uses setAccessible to hack into the private constructor of java.lang.invoke.MethodHandles.Lookup will be disappointed for example.
This change will expose hacks in many existing libraries and tools. As a workaround then a new command line option `--add-opens` can be used to open specific packages for "deep reflection". For example, a really popular build tool fails with this refresh because it uses setAccessible + core reflection to hack into a private field of an unmodifiable collection so that it can mutate it, facepalm! This code will continue to work as before when run with `--add-opens java.base/java.util=ALL-UNNAMED` to open the package java.util in module java.base to "all unnamed modules" (think class path).
Any help reporting issues to popular tools and libraries would be appreciated.
A debugging aid that is useful to identify issues is to run with -Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when setAccessible fails, this is particularly useful when code swallows exceptions without any logging.
Rgds,Rory
[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html <http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html>
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland
RE: JDK 9 b148 including a refresh of the module system is available on java.net
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
build 148 broke the unmap hack. The problem is an additional bug in Java: The unmap hack should be more unspecific on the Exception Type (InaccessibleObjectException does not extend ReflectiveOperationException). I will change this in an issue, but the general problem persists:
With the buil 148 version we can no longer unmap byte buffers:
java.lang.reflect.InaccessibleObjectException: Unable to make public jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() accessible: module java.base does not "opens java.nio" to unnamed module @10b73cf
[junit4] > at java.base/jdk.internal.reflect.Reflection.throwInaccessibleObjectException(Reflection.java:427)
[junit4] > at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:201)
[junit4] > at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:192)
[junit4] > at java.base/java.lang.reflect.Method.setAccessible(Method.java:186)
[junit4] > at org.apache.lucene.store.MMapDirectory.unmapHackImpl(MMapDirectory.java:345)
[junit4] > at java.base/java.security.AccessController.doPrivileged(Native Method)
[junit4] > at org.apache.lucene.store.MMapDirectory.<clinit>(MMapDirectory.java:326)
[junit4] > ... 42 moreThrowable #2: java.lang.NullPointerException
[junit4] > at org.apache.lucene.search.TestDateSort.tearDown(TestDateSort.java:71)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:538)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de <http://www.thetaphi.de/>
eMail: uwe@thetaphi.de
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Friday, December 9, 2016 4:35 PM
To: 'Rory O'Donnell' <ro...@oracle.com>
Cc: 'Dalibor Topic' <da...@oracle.com>; balchandra.vaidya@oracle.com; abdul.kolarkunnu@oracle.com; 'Dawid Weiss' <da...@cs.put.poznan.pl>; dev@lucene.apache.org
Subject: RE: JDK 9 b148 including a refresh of the module system is available on java.net
Hi Rory,
thanks for the message! I put build 148 in the Jenkins test rotation. Let’s see what happens.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de <http://www.thetaphi.de/>
eMail: <ma...@thetaphi.de> uwe@thetaphi.de
From: Rory O'Donnell [ <ma...@oracle.com> mailto:rory.odonnell@oracle.com]
Sent: Friday, December 9, 2016 11:34 AM
To: <ma...@thetaphi.de> uwe@thetaphi.de
Cc: <ma...@oracle.com> rory.odonnell@oracle.com; Dalibor Topic < <ma...@oracle.com> dalibor.topic@oracle.com>; <ma...@oracle.com> balchandra.vaidya@oracle.com; <ma...@oracle.com> abdul.kolarkunnu@oracle.com; Dawid Weiss < <ma...@cs.put.poznan.pl> dawid.weiss@cs.put.poznan.pl>; <ma...@lucene.apache.org> dev@lucene.apache.org
Subject: JDK 9 b148 including a refresh of the module system is available on java.net
Hi Uwe & Dawid,
JDK 9 build b148 <https://jdk9.java.net/download/> includes an important Refresh of the module system [1] , summary of changes are listed here <http://download.java.net/java/jdk9/changes/jdk-9+148.html> .
This refresh includes a disruptive change that is important to understand.
For those that have been trying out modules with regular JDK 9 builds then be aware that `requires public` changes to `requires transitive`. In addition, the binary representation of the module declaration (module-info.class) has changed so that you need to recompile any modules that were compiled with previous JDK 9 builds.
As things stand today in JDK 9 then you use setAccessible to break into non-public elements of any type in exported packages. However, it cannot be used to break into any type in non-exported package. The current specified behavior was a compromise for the initial integration of the module system. It is of course not very satisfactory, hence the #AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With the updated proposal in the JSR, this refresh changes setAccessible further so that it cannot be used to break into non-public types, or non-public elements of public types, in exported packages. Code that uses setAccessible to hack into the private constructor of java.lang.invoke.MethodHandles.Lookup will be disappointed for example.
This change will expose hacks in many existing libraries and tools. As a workaround then a new command line option `--add-opens` can be used to open specific packages for "deep reflection". For example, a really popular build tool fails with this refresh because it uses setAccessible + core reflection to hack into a private field of an unmodifiable collection so that it can mutate it, facepalm! This code will continue to work as before when run with `--add-opens java.base/java.util=ALL-UNNAMED` to open the package java.util in module java.base to "all unnamed modules" (think class path).
Any help reporting issues to popular tools and libraries would be appreciated.
A debugging aid that is useful to identify issues is to run with -Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when setAccessible fails, this is particularly useful when code swallows exceptions without any logging.
Rgds,Rory
[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html <http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html>
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland
RE: JDK 9 b148 including a refresh of the module system is available on java.net
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Rory,
thanks for the message! I put build 148 in the Jenkins test rotation. Let’s see what happens.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de <http://www.thetaphi.de/>
eMail: uwe@thetaphi.de
From: Rory O'Donnell [mailto:rory.odonnell@oracle.com]
Sent: Friday, December 9, 2016 11:34 AM
To: uwe@thetaphi.de
Cc: rory.odonnell@oracle.com; Dalibor Topic <da...@oracle.com>; balchandra.vaidya@oracle.com; abdul.kolarkunnu@oracle.com; Dawid Weiss <da...@cs.put.poznan.pl>; dev@lucene.apache.org
Subject: JDK 9 b148 including a refresh of the module system is available on java.net
Hi Uwe & Dawid,
JDK 9 build b148 <https://jdk9.java.net/download/> includes an important Refresh of the module system [1] , summary of changes are listed here <http://download.java.net/java/jdk9/changes/jdk-9+148.html> .
This refresh includes a disruptive change that is important to understand.
For those that have been trying out modules with regular JDK 9 builds then be aware that `requires public` changes to `requires transitive`. In addition, the binary representation of the module declaration (module-info.class) has changed so that you need to recompile any modules that were compiled with previous JDK 9 builds.
As things stand today in JDK 9 then you use setAccessible to break into non-public elements of any type in exported packages. However, it cannot be used to break into any type in non-exported package. The current specified behavior was a compromise for the initial integration of the module system. It is of course not very satisfactory, hence the #AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With the updated proposal in the JSR, this refresh changes setAccessible further so that it cannot be used to break into non-public types, or non-public elements of public types, in exported packages. Code that uses setAccessible to hack into the private constructor of java.lang.invoke.MethodHandles.Lookup will be disappointed for example.
This change will expose hacks in many existing libraries and tools. As a workaround then a new command line option `--add-opens` can be used to open specific packages for "deep reflection". For example, a really popular build tool fails with this refresh because it uses setAccessible + core reflection to hack into a private field of an unmodifiable collection so that it can mutate it, facepalm! This code will continue to work as before when run with `--add-opens java.base/java.util=ALL-UNNAMED` to open the package java.util in module java.base to "all unnamed modules" (think class path).
Any help reporting issues to popular tools and libraries would be appreciated.
A debugging aid that is useful to identify issues is to run with -Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when setAccessible fails, this is particularly useful when code swallows exceptions without any logging.
Rgds,Rory
[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-November/005276.html <http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html>
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland