Status Java 9 Jigsaw module system

[Uwe Schindler <uw...@thetaphi.de>]

Hi,

Robert and I spent a lot of time to get Lucene + Solr running with Java 9 EA builds that have the new Jigsaw module system enabled. You can download them at:
https://jdk9.java.net/jigsaw/

Basically Lucene is now working 100% correctly. Solr is also working mostly, but the are some test failures left. I ran Solr tests under Windows, so not all may be covered, it would be good if somebody with Linux could run the tests, too.

The following problems occur with Jigsaw and because this affects also external libraries, this may be hard to fix for Solr:
- It is no longer possible to forcefully make internal Java runtime classes accessible. So you cannot, for example, get some security class from the runtime and because you are missing a feature, just make a method visible and call it. This will always throw a new type of Exception: java.lang.reflect.InaccessibleObjectException
- Only some APIs are excluded by that: sun.misc.Cleaner and sun.misc.Unsafe (JEP 260; http://openjdk.java.net/jeps/260). This help for forcefully unmapping MMapDirectories bytebuffer. Although they try to fix the unmapping problem now!
- Robert and I added a new forbidden-apis rule to disallow AccessibleObject#setAccessible anywhere in code, also tests. For the legitimate uses (see MMapDirectory), you can suppress forbidden.
- I fixed many tests in Lucene+Solr to not reflect anymore by just making some helper classes or change access rights. Sometimes the setAccessible was just outdated. I also fixed Solr's SystemInfoHandler to work according to specs. The side effect: it also works on IBM J9 :-)
- We had to fix some tests, because also RamUsageTester is affected by the issue. If a test leaves static references to JDK-internal classes, the RAM usage can no longer be determined. It will throw above exception on test suite tear down. So if you see tear-down errors with InaccessibleObjectException, check all static fields in your test and null them on @AfterClass.

The test failures left in Solr (on Windows are):
- org.apache.solr.cloud.SaslZkACLProviderTest.testSaslZkACLProvider

   [junit4]    > Throwable #1: java.lang.RuntimeException: java.lang.IllegalAccessException: Class org.apache.hadoop.minikdc.MiniKdc can not access a member of class sun.security.krb5.Config (module java.security.jgss) with modifiers "public static", module java.security.jgss does not export sun.security.krb5 to <unnamed module @6d2a209c>
   [junit4]    >        at org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.run(SaslZkACLProviderTest.java:211)
   [junit4]    >        at org.apache.solr.cloud.SaslZkACLProviderTest.setUp(SaslZkACLProviderTest.java:81)
   [junit4]    >        at java.lang.Thread.run(java.base@9.0/Thread.java:746)
   [junit4]    > Caused by: java.lang.IllegalAccessException: Class org.apache.hadoop.minikdc.MiniKdc can not access a member of class sun.security.krb5.Config (module java.security.jgss) with modifiers "public static", module java.security.jgss does not export sun.security.krb5 to <unnamed module @6d2a209c>
   [junit4]    >        at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(java.base@9.0/AccessibleObject.java:384)
   [junit4]    >        at java.lang.reflect.AccessibleObject.checkAccess(java.base@9.0/AccessibleObject.java:376)
   [junit4]    >        at org.apache.hadoop.minikdc.MiniKdc.initKDCServer(MiniKdc.java:478)
   [junit4]    >        at org.apache.hadoop.minikdc.MiniKdc.start(MiniKdc.java:320)
   [junit4]    >        at org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.run(SaslZkACLProviderTest.java:204)
   [junit4]    >        ... 38 moreThrowable #2: java.lang.NullPointerException
   [junit4]    >        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.shutdown(ZkTestServer.java:334)
   [junit4]    >        at org.apache.solr.cloud.ZkTestServer.shutdown(ZkTestServer.java:526)
   [junit4]    >        at org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.shutdown(SaslZkACLProviderTest.java:218)
   [junit4]    >        at org.apache.solr.cloud.SaslZkACLProviderTest.tearDown(SaslZkACLProviderTest.java:116)
   [junit4]    >        at java.lang.Thread.run(java.base@9.0/Thread.java:746)

This is really bad, bad, bad! All security related stuff should never ever be reflected on! So we have to open issue in MiniKdc project so they remove the "hacks". Elasticsearch had similar problems with Amazon's AWS API. The worked around with a funny hack in their SecurityPolicy (https://github.com/elastic/elasticsearch/pull/13538). But as Solr does not run with SecurityManager in production, there is no way to do that. We should make clear to such projects that reflecting on Java runtime classes is now a no-go (it was always a no-go, but especially security related projects like bouncycastle do this, it looks to me that they completely counter their own improvements by that).

- The SolrMorphlinesTest breaks with some strange error. No idea, but only happens with Jigsaw.

My general comment: The morphlies and map-reduce packages look completely unmaintained to me. I see no reason to have them bundled with Solr, especially as they are shipped in separate. Noble Paul said the same a while back. I would really like to let them go. It has nothing to do with Solr as a server, it is just some DIH-like dataimporter that could also live outside.

I will open a bug report for the MiniKdc issue. But happily Lucene + Solr will run fine with Java 9 Jigsaw after Version 5.4.

Happy coding,
Uwe + Robert

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org